Text-synchronized media utilization and manipulation based on an embedded barcode

ABSTRACT

The embodiments relate to using barcodes embedded in a text document, such as a transcript. The barcode comprises a reference number or reference to a synchronization index associated with multimedia. When the barcode is read by a mobile computing device, the barcode indicates a location of multimedia that corresponds to the relevant portion of the transcript in which it appears. Methods and systems are also provided for controlling delivery of a text document based on a barcode. A mobile computing device scans a barcode or receives a portion of the text document and recognizes that a barcode has been embedded in the document. The barcode provides an address to a first computer by which the mobile computing device can authenticate itself and gain access to the text document. Based on the authentication, the mobile computing device may then obtain the text document as well as associated multimedia.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 13/342,865, entitled “Text-Synchronized Media Utilization andManipulation,” which claims priority to U.S. Provisional PatentApplication Ser. No. 61/429,399, entitled “Text-Synchronized MediaUtilization and Manipulation,” which is herein incorporated by referencein its entirety.

BACKGROUND

Today, multimedia content can be recorded, displayed, and accessed on awide variety of platforms including mobile computing devices, such aslaptops, tablet computers, smart phones, etc. In general, multimedia maycomprise a combination of text, audio, images, animation, video, orinteractive content.

Unfortunately, it remains difficult to encode and synchronize text thatis associated with a multimedia file. For example, the judicial systemrequires production and review of a substantial amount of informationincluding audio and visual evidence, document production and review ofwritten and electronically generated information, and testimony ofvarious involved parties. Testimony is understood to be a process bywhich an attorney asks oral questions of a witness and the witnessprovides answers under oath. A word-for-word record of the questionsasked and the answers given is called the testimony transcript.Testimony is normally taken and recorded during a deposition or a trial.During a trial or deposition, testimony is recorded by a court reporterusing a special typewriter-like device normally referred to as astenograph machine. The output of the stenograph machine is a long papertape of printed phonetic characters capturing, in a form of short hand,the word-for-word record of the testimony. The court reporter uses thepaper tape to create a text translation of the tape or text data. Thistext data is created using a typewriter or a word processor. The textdata is referred to as a transcript of the testimony and is historicallycreated “off-line” from the actual testimony.

During the discovery and research phase of litigation, the attorney willsearch through testimony for key statements made by a witness. Theattorney uses software tools for searching and annotating the ASCIItestimony or text data.

A shortcoming of some transcript management utilities is that theyoperate on a full version of Microsoft Windows operating system, whichplaces large demands on processor speed and RAM memory. As a result, thefull Microsoft Windows operating system is unsuitable for mobilecomputing devices, and they cannot run transcript management utilities.The result is that there has been no transcript management utility fordisplay of smoothly scrolling, synchronized text and multimedia for useon a mobile computing device, where the user may perform a gesture onthe text to cause the multimedia to jump to a desired location and beginplayback.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee. Systems and methods which embody the variousfeatures of the invention will now be described with reference to thefollowing drawings, in which:

FIG. 1 is a representation of the viewing screen layout of a mobilecomputing device, partitioned into several areas.

FIG. 2 illustrates an example of a movie being output using the viewingscreen layout of a mobile computing device shown in FIG. 1.

FIG. 3 illustrates another example representation of the viewing screenlayout of a mobile computing device, partitioned into several areas.

FIG. 4 illustrates an example of a deposition video and transcript beingoutput using the viewing screen layout of a mobile computing deviceshown in FIG. 3.

FIG. 5 illustrates an example representation of the viewing screenlayout of a mobile computing device, partitioned into several areas.

FIG. 6 is a schematic of elements of an example system.

FIG. 7 illustrates an example interface that may be used in handlingerrata sheet operations.

FIG. 8 illustrates an example of merging designations.

FIG. 9 illustrates example data structures that may be used in mergingdesignations.

FIG. 10 illustrates an example interface for assigning issues todesignations.

FIGS. 11A-C illustrates an example interface where each party has itsown work pane display of a transcript.

FIG. 12 illustrates an example printed page output in a portrait view.

FIGS. 13A-B illustrates an example printed page output showing onlydesignated portions of a transcript.

FIG. 14 illustrates an example printed page output in a landscape view.

FIG. 15 illustrates an example of a manuscript printing showinghighlighted designations.

FIG. 16 illustrates an example interface for annotating evidentiaryrulings on objections.

FIG. 17 illustrates an example format for a synchronization index.

FIG. 18 illustrates an example of a linked exhibit.

FIG. 19 illustrates exemplary graphical elements may be displayed toindicate other highlighted/designated portions of a text transcript.

FIG. 20 illustrates a display screen of a mobile device that can bedivided into multiple sections.

FIG. 21 illustrates an exemplary screen shot of a mobile device for atranscript synchronized to a multimedia audio file.

FIG. 22 illustrates an exemplary screen shot of a mobile device playingmultimedia content and a transcript for education material from anorganization and a listing of other courses with multimedia content.

FIG. 23 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript.

FIG. 24 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript accompanied with a comments section onthe display.

FIG. 25 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript accompanied with a section of thedisplay for showing errata and a thumbnail of an attachment to thetranscript.

FIG. 26 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a pop-up window showing a linkedexhibit.

FIG. 27 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a section for exhibits linked tothe deposition.

FIG. 28 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a section showing objections toportions of the testimony by the deponent.

FIG. 29 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a summary section on the display.

FIG. 30 illustrates an exemplary screen shot of a mobile device for afiltered deposition video and transcript with a summary section on thedisplay.

FIG. 31 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a section listing witnesses.

FIG. 32 illustrates an exemplary screen shot of a mobile device playinga news story having a video clip and corresponding text.

FIG. 33 illustrates an exemplary screen shot of a mobile device playingmultimedia content with a transcript and a section listing other filesavailable to the user.

FIG. 34 illustrates an exemplary screen shot of a mobile device playingmultimedia content with a transcript from an educational institution.

FIG. 35 illustrates an exemplary screen shot of a mobile device playingmultimedia content with a transcript from a musical group and sectionlisting other available content from the musical group.

FIG. 36 illustrates an exemplary screen shot of a mobile device playingmultimedia content and a transcript for an audio book.

FIG. 37 illustrates an exemplary screen shot of a mobile device playinga movie and a transcript with a section for navigating through themovie's content.

FIG. 38 illustrates an exemplary layout of a display of a mobile devicefor brand tagging in multimedia content.

FIG. 39 illustrates an exemplary data structure for the synchronizationindex.

FIG. 40 illustrates an issue-coded transcript comprising barcode placedin the margin.

FIG. 41 illustrates a process flow chart for creating a barcodeassociated with a portion of a transcript and multimedia.

FIG. 42 illustrates a software user interface for displaying barcodeassociated with a portion of a transcript and multimedia.

FIG. 43 illustrates a multimedia display with barcode for retrieval ofmultimedia with a mobile computing device.

FIG. 44 illustrates a program guide with barcode for retrieval ofmultimedia with a mobile computing device.

SUMMARY

In one embodiment, methods and systems are provided for using barcodesembedded in a text document, such as a transcript. The barcode comprisesa reference number or reference to a synchronization index associatedwith multimedia. When a mobile computing device reads the barcode, thebarcode indicates a location of multimedia that corresponds to therelevant portion of the transcript in which it appears.

In another embodiment, methods and systems are provided for controllingdelivery of a text document based on a barcode. A mobile computingdevice receives a portion of the text document and recognizes that abarcode has been embedded in the document. The barcode provides anaddress to a first computer by which the mobile computing device canauthenticate itself and gain access to the text document. Based on theauthentication, the mobile computing device may then obtain the textdocument as well as associated multimedia.

DETAILED DESCRIPTION

Techniques are described for deploying a synchronization index andassociated media (e.g., multimedia), for manipulations to thesynchronization index, for navigation between the synchronization indexand associated media (e.g., multimedia), and for synchronous playback ofmedia (e.g., multimedia) and associated text, preferably using a mobilecomputing device. For example, a system may use multimedia with asynchronized text transcription thereof, or a synchronization index, ona mobile computing device for viewing and manipulating the multimedia,using the synchronized text transcription. In this example, the mobilecomputing device may be used to navigate the multimedia, annotate thetext transcription, achieve synchronous play of the multimedia, andcommunicate to a computer other than the mobile computing device (e.g.,a remote server) manipulations performed on the text or multimedia.

In one aspect, a system enables viewing and manipulating synchronizedmultimedia and text data stored on a remote server. The system may usemultimedia with a synchronized text transcription thereof, or asynchronization index, on a mobile computing device for viewing andmanipulating the multimedia, using the synchronized text transcriptionto navigate the multimedia and vice versa, optionally annotating thetext transcription, and communicating to a computer other than themobile computing device manipulations performed on the text ormultimedia.

More generally, in some implementations, a system enables viewing andmanipulating multimedia and text data. Specifically, multimedia and anassociated synchronization index are communicated to a remote server.The remote server operates to re-format the synchronization index andcommunicate it to a mobile computing device. With the mobile computingdevice, its software, and the synchronization index, the user may thenperform manipulations on the text and control the multimedia delivery tothe mobile computing device, or to a separate receiving device. A userhaving performed manipulations to the synchronization index may thencommunicate those manipulations back to the remote server to allowsubsequent users to retrieve the synchronization index for use withtheir own mobile computing devices, thereby realizing a collaborativeworking environment.

More recently, stenograph machines and computer software programs havebeen developed which translate the phonetic characters while thestenograph operator is typing. The output of this automated translationis a stream of ASCII characters or text data that is stored on a disk,data card (or other portable storage device), or in a personal computerattached directly to the stenograph machine by a serial communicationslink. These types of stenograph machines and the associated computer andsoftware are referred to as Computer-Aided Transcription or CAT systems.The text data output of the CAT system is an ASCII text computer file ofthe testimony. The court reporter edits the ASCII file to create acompleted version of the transcript, which can be viewed on a computeror can be printed. Typically, the court reporter then forwards thecompleted version of the transcript to a court reporting firm forquality review; standardization of formatting, to include page and linenumbers, headers and/or footers, information to identify the courtreporting agency and provide contact information, certification pages,and so on; and production and distribution to the attorneys and/or thecourt. The transcript is then a “final” transcript, and it may beformatted in a number of formats for litigation related softwareapplications and printing.

Along with the development of CAT systems, video capture devices haveadvanced the art of making a record of testimony, most commonly employedas part of the deposition process. In major and/or complex litigationcases, testimony from depositions and/or a trial is video recorded inaddition to having a transcript created. The person performing the videorecording is called a videographer. The videographer uses a standard,commercially available video capture device. Video recording proceedssimultaneously with the creation of the written record of the testimonyby the court reporter. After the testimony is complete, the lawyer willnormally ask the court reporter for a copy of the text data or ASCIItext file of the testimony and will ask the videographer for a copy ofthe video data of the testimony.

The video recording has historically been recorded to a tape media,which videotapes are commercially available in a number of sizes andformats. The videographer typically later encodes the content into anelectronic format with commercially available playback and encodingequipment. A common encoding format for the legal industry is MPEG-1, asMPEG-1 functions with substantial stability with commercially availabletrial presentation software applications.

Transcript Management

A lawyer uses the ASCII file for many purposes including review andresearch of the deposition testimony, preparation for trial, andresearch prior to cross examination during trial. A variety of softwaretools have been developed which, among other things, allow a lawyer toelectronically search testimony text for key words, to annotate, toinsert page marks, to associate certain portions of testimony withissues and to create printouts of testimony for insertion in pleadings,submissions to the court, and trial presentations.

During the discovery and research phase of litigation, the attorney willsearch through testimony for key statements made by a witness. Theattorney uses software tools for searching and annotating the ASCIItestimony or text data.

A shortcoming of some transcript management utilities is that theyoperate on a full version of Microsoft Windows operating system, whichplaces large demands on processor speed and RAM memory. As a result, thefull Microsoft Windows operating system is unsuitable for mobilecomputing devices, and they cannot run transcript management utilities.The result is that there has been no transcript management utility fordisplay of smoothly scrolling, synchronized text and multimedia for useon a mobile computing device, where the user may perform a gesture onthe text to cause the multimedia to jump to a desired location and beginplayback.

To understand some problems associated with transcript management, it isimportant to understand how a team of attorneys collaboratively works ona transcript. First, a team of attorneys reviews a transcript internallywithin the law firm or legal department. This internal review may bedone by several attorneys, by a junior associate who has his workchecked by a more senior attorney, by a paralegal, or in conjunctionwith the legal client or corporate representative who may have knowledgeabout the testimony or an interest in the litigation. The purpose of theinternal review is to determine what questions and answers may be usefuleither in upcoming depositions, during settlement negotiations, for usein legal briefs, or at trial. Internal review is where “issue codes” areimportant. The term issue code refers generally to a range of wordsselected within a transcript and labeled to correspond to an issuewithin the litigation.

The circumstance may exist after a deposition where a paralegal, orother professional, may be asked to review the transcript and toannotate it for the law firm partner who is preparing to depose anotherwitness, say, an expert witness. Accordingly, the partner would ask theparalegal to “issue code” all sections of the transcript relating to thepartner's anticipated line of questioning of the new witness, forexample, to explore inconsistencies from one witness to another. Theparalegal does the review, and issue codes several parts of thetranscript, say, in yellow. Before the deposition, the partner will comeback to the transcript and read, as quickly and efficiently as possible,only those yellow sections. This process may occur for several issues,each of which may overlap and be a different color. The transcriptitself can, if all issues are viewed simultaneously, become too busy toread, as overlapping colors bleed into one another, the number of colorsbecome distracting as the numbers of issues increase, and so forth.

The goal of a legal team before trial is to piece together a coherentstory using very discrete deposition designations, or video clipscorresponding to deposition designations. These designations willsupport the credibility of legal briefs to the court, and such deponenttestimony is frequently cited and footnoted accordingly.

In transitioning to trial, the “issue codes” as above become onlysecondarily useful. At that point, the primary purpose of the transcriptmanagement utility is to select only the best material to use in court.Assume a deponent has died and cannot, obviously, testify on the witnessstand. The attorneys will be able to show his video recorded depositionin court, pursuant to the rules of evidence. The law firm will pare downthe deponent's testimony to the best segments possible, for example as ahighlight reel, leaving out the testimony that would tend to underminethe coherent story that they are trying to tell in court. This istypically a sub-set of the material already issue coded. Video run timeis often a key consideration, 1) because the court can allocate acertain number of hours for one side to present its entire case, in aneffort to get to the heart of the controversy and to respect the jury'stime, and 2) because too much video can be boring for the jury to watch,causing them to cease paying attention or fail to appreciate therelevance of the testimony to the overall dispute.

Accordingly, the plaintiff's attorneys have a time before trial by whichthey need to identify to the defendants, and the court, which portionsof testimony they intend to rely upon at trial. For this reason,plaintiff's attorneys will need to 1) shorten, modify, or eliminatetestimony they have already issue coded, 2) “flatten” the issue codesinto one common issue code, i.e. “Plaintiff's direct designations”, and3) transmit them to the defendant.

The defendant will then take “Plaintiff's direct designations” and addback all the material that would be helpful for defendant's case.Defendant's objectives are to 1) undermine the credibility of thewitness, 2) lessen the impact of the testimony designated by theopposing party, 3) confuse the jury, or 4) add back so much materialthat it becomes difficult for a jury to understand the importance of thetestimony as a whole or the reasons for which it is being proffered forthe jury's consideration. These would be “issue coded” as, say,“Defendant's counter designations.” This process may occur again withcounter-counter designations by each party.

The trial consultant, generally an individual with informationtechnology and trial presentation software expertise, is tasked withdisplaying with a computer in the courtroom only the proper videosegments. Other than to provide an accurate accounting of what video runtime is to be allocated against plaintiff's total allotted time anddefendant's allotted time, the trial consultant is indifferent as towhat clip “belongs” to whom. All of the designations, once finalized bythe attorneys, should be “flattened”, giving the consultant the fewestnumber of edit points to “fine tune.” Accordingly, there is one merge ofdesignations done after internal review when preparing for trial, andone merge done when a plaintiff and a defendant have absolutelyfinalized the designations and given them to the trial consultant.

Without the techniques described throughout this disclosure, transcriptmanagement and communication between parties and the court may be aprocess plagued with inefficiencies.

Timestamping a Transcript

In addition to having the written transcript organized and available tothe attorney whilst eliciting testimony at trial, the need frequentlyarises to present to the court and jury a specific part of thetestimony. When testimony has been video recorded, it is desirable tohave the ability to use the video associated with key statements toimpeach, to expose an inconsistency between a witness's in-courttestimony and earlier testimony, to show a witness's demeanor, and soforth. As an examining attorney, having access to a witness's videotapeddeposition, and having the witness know retrieval and display of priortestimony is a possibility, is also a strong psychological tool that canaffect a witness's performance on the witness stand.

To view the video, the attorney historically had to have access to aVCR, a DVD player, a video player or DVR device and either a TV screenor monitor. With the advent of trial presentation software, along withadvances in computing speed and storage capacity, attorneys and theirtrial consultants can link, or marry, the deposition transcript to thedeposition video. This process is commonly referred to in the legalindustry as “timestamping” a transcript.

The important job of synchronizing multimedia to a transcript or textdata may be done by a human operator (either by a law firm or a thirdparty vendor) using trial management software, or through a third partyvendor who utilizes voice recognition software, or a combination ofvoice recognition software with a human quality review check. Othermethods may be employed for creating closed captioned and/or subtitledoutputs for use by the hearing impaired, and the multiplicity of methodsto create synchronization indexes enhances the number of potential usesfor the techniques described throughout this disclosure. Problems ariseas the parties move through the litigation process and are tasked withsynchronizing video data and text data, and they are further burdenedwith using disparate software applications to accomplish issue coding,annotating, editing, time stamping, video clip creation, video clipreview, report creation to detail video clip duration, errata sheettracking, and distribution of content to opposing parties, clients, andthe court.

Regarding timestamping a transcript, one day of deposition testimony forone deponent is almost always comprised of more than one video segment.One video segment generally corresponds to one videotape thevideographer fills during the deposition and later encodes to a singleMPEG-1 file, the legal industry standard format for video encoding.Depending on videotape stock and format, one video tape will hold aboutan hour of content, and most often no more than two hours of content,while most depositions run longer than two hours. There is typicallyonly one transcript for each deposition. An attorney needs to be able toassociate the single transcript to the multiplicity of video files.

For example, it is common within the court reporting and legalvideography industry for a 3-hour deposition of a deponent to be splitbetween three videotapes, each an hour in duration, that would beencoded as x.mpg for time-code zero to hour 1, x1.mpg for hour 1 to hour2, and x2.mpg for hour 2 to 3. Within some trial management utilities,x.mpg, x1.mpg, and x2.mpg are sequenced, for a 3 hour total duration,and associated with the single transcript. Using a trial managementutility, the user reads the transcript while the video plays and pressesa key when the last word on a line is spoken to have the software notethe time at which the input was made and note it in a table adjacent tothe line of testimony.

In some trial management utilities, the resultant timestamped file, orsynchronization index, is a Microsoft Access formatted database file, in.mdb format, with a column of timecode run duration that runssequentially from hour zero to hour three. Accordingly, a user of atrial management utility wishing to have the software call a line oftestimony at timecode 02:30 would be pointed to 00:30 of x2.mpg. Callingtestimony at time-code 01:45 would call x1.mpg at 00:45, and so on. Inthis fashion, the timestamped transcript, or synchronization index,allows the attorney to use the transcript as a roadmap, while thesoftware seeks first the correct video file, then seeks to the correctlocation within the video file, then displays each in congruent,synchronous fashion by refreshing the display of the transcript as thetimecode associated with each new transcript line is reached.

FIG. 39 illustrates an exemplary data structure for the synchronizationindex. As shown, the synchronization index comprises entries for eachline having a predetermined number of characters, such as 256characters, and each page may have a predetermined number of lines. Foreach line, the synchronization index indicates a timestamp andcoordinate for each character. The synchronization index assists insynchronizing multimedia content, such as audio and/or video clips, withtranscripts of text.

In some cases, it has been very difficult for users to create videoclips or audio clips in software, especially with known trialpresentation software, that correspond to issue codes created withintranscript management software. Users have been faced with amultiplicity of software applications, the combination of which aredifficult to learn and use, expensive to purchase, and impossible tojointly manage, particularly on mobile computing devices.

Trial Preparation

Trial presentation software applications have become an indispensibletool to today's litigators. These applications allow organization, quickaccess to and broadcast of all types of material the litigator needs attrial, including trial exhibits, demonstrative exhibits, multimedia,animations, transcripts, video testimony, timelines, images and so on.Most trial presentation utilities are characterized by running only on acomputer using a full version of the Microsoft Windows operating system(which itself requires a computer comprising at least a 233 MHzprocessor, 64 MB of RAM, and 1.5 GB of available hard drive space), byrequiring extensive training and practice to achieve an acceptable levelof competence, by costing hundreds of dollars to purchase or rent, bynecessity of payment of yearly maintenance fees to maintain an activelicense and access to technical support, and by frequently requiring apaid litigation consultant to operate it competently in a courtroomenvironment. Each, and the combination, is replete with a number ofshortcomings effecting the efficiency of litigation practice.

Transcript Errata Sheets

To illustrate just one of the problems associated with transcriptmanagement, one may look to the errata sheet used in litigation. TheFederal Rules of Evidence (Rule 30), and other local rules patternedafter them, allow a period of time after a deposition for the deponentto review the written transcript for errors before it is signed andreturned as an accurate transcription of the testimony. Commonly, thewitness will read through the transcript, note what he believes to beerrors, and note them in what is referred to in the legal industry as an“errata sheet.” A copy of this errata sheet is signed by the witness,returned to the court reporter or opposing counsel, and physicallyaffixed to the hardcopy of the official transcript as an appendix. Thus,the witness's signature is an attestation to the accuracy andcompleteness of the court reporter's transcription which, if all erratadenoted are remedied, would comprise the final, official transcript tobe submitted to the court. Accordingly, there historically has been nonotation in the original transcript at the point where a correction wasmade that would refer the reader to the correction in the errata sheet.

Moreover, and perhaps more problematically, content on an errata sheetis never added to or affixed to the electronic copy of a transcript,which is often in a .txt format. It is this .txt formatted transcriptthat is loaded into commercially available trial presentation software.For example, if a .txt transcript is loaded into a trial managementutility, the trial management utility creates a Microsoft Accessdatabase table (.mdb format) formatted to take one line of text andplaces it in one of the database cells. This database, and the contentof the cells, is called upon by the trial management utility whenvideotaped deposition testimony is played to give the appearance ofsynchronized, scrolling text beneath a video image. Since thecorrections of the errata sheet never make their way into the .txttranscript, they never make their way into the trial presentationsoftware, and they are never displayed, as they should be, as part ofthe synchronized, scrolling text beneath the video image.

The disparity between words clearly audible from the videotape and thewritten transcript is often noticed, if at all, on the eve of trial, bythe trial consultant. Text that does not match the video recording isproblematic because, according to procedural rules, it is the writtentranscript that is the official record and it is that record that ispassed on to appellate courts and preserved in the record. It can beproblematic and confusing if a jury is shown video testimony that doesnot exactly match the scrolling text that may be displayed with it. Itcan be further problematic and confusing, particularly to an appellatecourt, if a jury is shown video testimony without any scrollingtranscript, where the written transcript may, in fact, contradict thewords spoken on the video.

Prior to the techniques described throughout this disclosure, there hasbeen no systemic way to ensure errata sheet corrections functioned inconcert with trial presentation software.

Certain Closed Captioning, Subtitling and Historic Uses of Text

Closed captioning text and karaoke-type subtitles may present a numberof shortcomings emblematic of the art before the techniques describedthroughout this disclosure. First, closed captioned text is configuredto display over a part of the video screen, thereby obscuring at least apart of the video display itself. Second, closed captioned text oftenappears as text with a background graphic, for example an opaque orsemi-opaque box layered between the video and the individual textcharacters. As a consequence, more of the video display area is coveredthan would be the case if the individual characters were alonedisplayed. Third, a user in general can only select to have closedcaptions “on” (or displayed), or “off” (not displayed.) Fourth, a usercannot generally select text display font, color, or size. Fifth, a usercannot perform a text search. Sixth, a user cannot view text that doesnot correspond to the portion of video being viewed, that is, they areinextricably linked and cannot be separated apart from turning the textdisplay “off.” Seventh, the text cannot be highlighted, annotated,bookmarked, associated with a note, nor can any of the above be used tonavigate within the video. Eighth, a user cannot provide a link to thetext to a second user, intending for the second user to begin videoplayback at a time certain based on the linked text. Ninth, closedcaptioning is not optimized for use with mobile computing devices, ordisplay screens of relatively small size. Tenth, closed captioning textdata is difficult to alter, as it is transmitted as an integral part ofa video signal. In the case of “open subtitles” the subtitles arepermanently placed into the video frame itself and cannot be removed.

Eleventh, another pitfall of closed captioning occurs as a result of alimited number of text words being able to be displayed at once, whereasin certain circumstances a more complete text transcript would be morepreferable. For instance, song lyrics have the characteristics ofpoetry, or a complete story. The context of the words within the overallcomposition may have meaning to the artists, or to the reader, when readas a whole, rather than one line at a time or a few words at a time.Twelfth, closed captions are most often employed by the hearing impairedor in environments where audio for video is not readably audible, forexample in a restaurant or bar. Consequently, the vast majority of userswho consume multimedia content do not seek to utilize closed captionedtext even when it is made available, and many may not even be aware thatit is available as an option.

Thirteenth, closed captioning text and subtitles are not “interactive,”in that a user cannot do anything with the words other than watch themappear on screen when the video plays. The user cannot move ahead orbackward in the transcription, and the user cannot read any words otherthan the limited number of lines than can fit on the video screen,obscuring part of the screen. Closed captioning and karaoke-typesubtitles, and the information to display them in a synchronizationindex-type format, are characterized by passive non-interactivity,meaning there is no way for an operator to utilize the text in any wayother than watching it appear and disappear, typically from themultimedia viewing area itself. The user has no opportunity to use thewords in any fashion other than to read them.

By contrast, the synchronization index and software suited for use on amobile computing device of the present disclosure is characterized byinteractivity, meaning that the user is able to use the synchronous textdisplay to accomplish another action, namely convenient navigationwithin the multimedia by using the text, or convenient navigation of thetext by using the multimedia. Other actions may also be provided, asdiscussed further below, including annotation of text and provision ofadvertising to a mobile computing device. In the present disclosure, anenhanced overall user experience may be provided, where the text can beused as an interactive tool to facilitate navigation withincorresponding media.

As is evident, transcript management for litigation is a daunting taskbecause of continual updating, modifying, editing, issue coding,synchronization to video, video clip creation and editing, and need tocommunicate with the court and adversarial litigants, compounded by themultiplicity of complicated, expensive, computer-resource intensivesoftware applications required to accomplish each task.

The techniques described throughout this disclosure may address andsolve many of these shortcomings, for example by providing intuitivesystems and methods to manage a synchronization index and associatedmultimedia with a mobile computing device.

Despite the numerous systems and methods known for manipulatingtranscripts, synchronizing text to video, and presenting multimedia inthe courtroom, there is still a need for a system and methods tosimplify the transcript management process by providing asynchronization index and associated multimedia to a mobile computingdevice, being in communication with a hosting server and multimediaserver. In the present disclosure, a mobile computing device, with auser friendly interface, may be employed to simplify what has been acomplex, problematic workflow, to synchronously display text andmultimedia, to simplify navigation of multimedia, and to easecommunication among disparate users.

There is also a need for systems, products, software and methods tofacilitate use of mobile computing devices with synchronization indexesand corresponding multimedia for any type of multimedia that can beassociated, and manipulated, by corresponding text, such as music andlyrics, video/audio books, movies and screenplays, and other multimediarecordings.

The techniques described throughout this disclosure may address one ormore of these needs and may advantageously overcome one or moredeficiencies of other options. Certain embodiments of the inventionswill now be described. These embodiments are presented by way of exampleonly, and are not intended to limit the scope of the inventions. Indeed,the novel methods and systems described herein may be embodied in avariety of other forms. Furthermore, various omissions, substitutionsand changes in the form of the methods and systems described herein maybe made without departing from the spirit of the inventions. Toillustrate some of the embodiments, reference will now be made to thefigures.

FIG. 1 is a representation of the viewing screen layout of a mobilecomputing device, partitioned into several areas. In this example,computer source code (e.g., code written in Xcode or other suitableprogramming language) was compiled for mobile computing devices (e.g., atablet computer, such as the iPad, a mobile phone, such as the iPhone,etc.) running a mobile device operating system (e.g., Apple'siOS4.0/iOS4.2). Some terms used within this application may betrademarks owned by their respective owners. In the case of a tabletcomputer, the display screen may be divided into several main areas: themultifunction area 1, the header area 8, the media area 9, the text area10, and the scrub-bar area 11.

The media area 9 may display output of a media file selected by a user.For example, the media area 9 may display image data in the form of avideo or still image. In this example, the media area 9 may display adeposition video, a movie video, a music video, an illustration from anelectronic book, or other types of media a user may desire to output.The tablet computer also may output, using a speaker, audio thatcorresponds to the displayed media, such as audio of a deposition videobeing displayed, audio of a movie being displayed, audio of a musicvideo being displayed, audio of an electronic book for which anillustration is being displayed, etc.

The text area 10 may display a text transcript that corresponds to themedia being displayed in the media area 9. For example, the text area 10may display a text transcript of a deposition video being displayed, atext transcript of dialogue from a movie being displayed, a texttranscript of lyrics of a music video being displayed, a text transcriptof an electronic book for which an illustration is being displayed, etc.

The text area 10 may control display of the text transcript in asynchronized manner with the media being displayed in the media area 9.For instance, the tablet computer may identify a portion of the texttranscript that corresponds to the portion of the media being displayedin the media area 9 and display the identified portion of the texttranscript. The text area 10 also may include a highlight bar thatserves as a position indicator, for example, by highlighting the currentline of text being output as audio for the media displayed in the mediaarea 9. In some implementations, the tablet computer controls thehighlight bar to remain static and causes the text displayed in the textarea 10 to scroll up as each line of audio shown in the text transcriptis output. In other implementations, the tablet computer controls thehighlight bar to move down a line in the text transcript as each line ofaudio shown in the text transcript is output and controls the highlightbar to move back to the top of the text area 10 when it reaches thebottom of the text area 10 and a new portion (e.g., page) of the texttranscript is displayed in the text area 10. Other options for thehighlight bar may be employed and highlighting of each word in the texttranscript also may be used. The tablet computer may control display ofthe text transcript in a synchronized manner with the media beingdisplayed in the media area 9 by using a synchronization index asdescribed throughout this disclosure.

The text area 10 also may be manipulated by a user to control display ofthe media being displayed in the media area 9. For instance, the usermay select a line in the text transcript being displayed in the textarea 10 and the tablet computer may start display of the media at thepoint corresponding to the selected line in the text transcript. If theuser selects multiple lines of the text transcript, the tablet computermay start display of the media at the first selected line in the texttranscript and stop display of the media at the last selected line inthe text transcript. The text transcript displayed in the text area 10may be used to achieve other types of control of the media displayed inthe media area 9, as described throughout this disclosure.

In addition, in some implementations, the tablet computer may enable auser to independently review the text transcript displayed in the textarea 10. For instance, the tablet computer may enable a user to scrollthrough the text transcript with or without impacting the mediadisplayed in the media area 9. Also, the tablet computer may enable auser to search the text transcript to find portions of the texttranscript and corresponding media that the user would like to perceive.Further, the tablet computer may enable a user to manipulate the texttranscript by, for example, adding annotations to the text transcriptand designating portions of the text transcript of interest, asdescribed throughout this disclosure. The tablet computer may send themanipulations (e.g., annotations and designations) to a remote serverfor remote storage and to be synchronized with other systems the usermay use for annotating a text transcript and designating portions of thetext transcript.

The header area 8 may display information (e.g., metadata) about themedia being displayed in the media area 9 and the text being displayedin the text area 10. For instance, the header area 8 may display a titleof the media, an author or director of the media, identity of actors orcharacters shown in the media, or any other information (e.g., metadata)that may be useful in providing additional information of the media andtext being displayed.

The scrub-bar area 11 includes a scrub bar that may be used to controldisplay of the media displayed in the media area 9 and the textdisplayed in the text area 10. For example, as a user moves the scrubbar control back and forth, both the media displayed in the media area 9and the text displayed in the text area 10 may scroll in a synchronizedmanner.

The multifunction area 1 may include a media select icon 2, a searchicon 3, a title area 4, an issue icon 5, a text font display icon 6, andan annotation icon 7. The media select icon 2 may enable a user toselect media to display in the media area 9 by, for example, causingdisplay of a directory from which the user may select a desired mediafile or causing display of a list of available media files for userselection. The search icon 3 may enable a user to search the texttranscript displayed in the text area 10. The title area 4 may display atitle of the media being output and the text font display icon 6 mayenable control of font parameters of the text in the text area 10 isdisplayed. The issue icon 5 may enable a user to add an issue associatedwith any portion of the text transcript capable of being displayed inthe text area 10 and the annotation icon 7 may enable a user to add anannotation to the text transcript displayed in the text area 10.

FIG. 2 illustrates an example of a movie being output using the viewingscreen layout of a mobile computing device shown in FIG. 1. As shown inFIG. 2, the header area 8 displays a title of the movie, the director ofthe movie, the writer of the movie, the scene name of the scenecurrently displayed, and the names of the characters (and/or actors)having speaking roles in the scene. The scene name and the names of thecharacters (and/or actors) may dynamically change as the movie plays.The header area 8 also includes a display of a concept for the scenecurrently being displayed and an image of a concept sketch of the scenecurrently being displayed. The concept and concept sketch maydynamically change as the movie plays. In addition, the media area 9shows a display of the movie video and the text area 10 shows a portionof a text transcript of the dialogue of the movie video being displayed.As described throughout this disclosure, the mobile computing device maycontrol display of the text transcript in the text area 10 in asynchronized manner with the video being displayed in the media area 9.

FIG. 3 illustrates another example representation of the viewing screenlayout of a mobile computing device, partitioned into several areas. Asshown, the display screen may be divided into several main areas: themultifunction area 1, the header area 13, the media area 12, the textarea 14, and the scrub-bar area 11. The multifunction area 1 and thescrub-bar area 11 may have the same characteristics described above withrespect to FIG. 1. Despite different size and layout, the media area 12,the header area 13, and the text area 14 may operate similarly to themedia area 9, the header area 8, and the text area 10 described abovewith respect to FIG. 1.

FIG. 4 illustrates an example of a deposition video and transcript beingoutput using the viewing screen layout of a mobile computing deviceshown in FIG. 3. As shown in FIG. 4, the header area 13 displays a courtassociated with the deposition, a civil action number associated withthe deposition, a name of the deponent in the deposition, a location ofthe deposition, and the date/time of the deposition. In addition, themedia area 12 shows a display of the video of the deposition and thetext area 14 shows a portion of a text transcript of the dialogue of thedeposition being displayed. As described throughout this disclosure, themobile computing device may control display of the text transcript inthe text area 14 in a synchronized manner with the video being displayedin the media area 12.

FIG. 5 illustrates an example representation of the viewing screenlayout of a mobile computing device, partitioned into several areas. Asshown, the display screen may be used when the mobile computing deviceis oriented in a landscape orientation. The mobile computing device mayswitch between the layout shown in FIG. 5 and the layout shown in FIG. 1or FIG. 3 depending on whether the mobile computing device detects thatit is being held or viewed in a landscape or portrait orientation.

In FIG. 5, the display screen may be divided into several main areas:the multifunction area 1, a media list area 17, the header area 19, themedia area 18, the text area 20, and the scrub-bar area 11. Themultifunction area 1 and the scrub-bar area 11 may have the samecharacteristics described above with respect to FIG. 1, except that themedia select icon 2 is displayed in the media list area 17. Despitedifferent size and layout, the media area 18, the header area 19, andthe text area 20 may operate similarly to the media area 9, the headerarea 8, and the text area 10 described above with respect to FIG. 1. Themedia list area 17 may display a directory or category name associatedwith the media files included in the media list area 17. The media listarea 17 may enable a user to quickly switch between different mediafiles by selecting a media file (e.g., Media File 1, Media File 2, MediaFile 3, Media File 4, or Media File N) included in the list. The medialist area 17 also includes an edit control 16 that enables a user toedit which media files are displayed in the list. For instance, the editcontrol 16 may be used to select a different directory or category ofmedia files to display in the list.

In one example, the display screen shown in FIG. 5 may be used todisplay music videos. In this example, the title area 4 may display aname of the song being output, the media area 18 may display a musicvideo of the song being output, and the text area 20 may display lyricsof the song being output in a synchronized manner with the music video.In addition, the header area 19 may display a release date for the song,a time length of the song, a label for the song, a writer of the song,and a producer of the song. The media list area 17 may display a name ofthe artist singing the song being output and include a list of othersongs for the artist that are available for selection.

In another example, the display screen shown in FIG. 5 may be used todisplay depositions. In this example, the title area 4 may display adeponent name, the media area 18 may display a video of the deposition,and the text area 20 may display a text transcript of the deposition ina synchronized manner with the deposition video. In addition, the headerarea 19 may display a court associated with the deposition, a civilaction number associated with the deposition, a name of the deponent inthe deposition, a location of the deposition, and the date/time of thedeposition. The media list area 17 may display a deponents category (ora cases category) and include a list of other depositions (or othercases) available for selection.

FIG. 6 is a schematic of elements of an example system. The systemcomprises: a hosting server 6-1; a multimedia server 6-2; a webapplication 6-3; text content 6-4; multimedia content 6-5; and at leastone mobile computing device 6-6. In most general terms, the webapplication is software (e.g., instructions executable by a processorand stored on a tangible, non-transitory computer-readable storagemedium) that may comprise user permissioning logic 6-7; file format anddata parsing logic 6-8; media delivery logic 6-9; mobile computingdevice function logic 6-10; search logic 6-11; annotation and edit logic6-12; video, display, and playlist logic 6-13; linked exhibit logic6-14; errata sheet logic 6-15; output formatting logic 6-16; schedulingand invoicing logic 6-17; and variable content logic 6-18. The webapplication, described in more detail below, may be software codeintended to function as a “cloud computing” application, softwarerunning on a server remote from users who access it and itsfunctionality by way of an internet connection, an internet or webbrowser, and a computer or, possibly, by a mobile computing devicerunning software specifically for communication with the webapplication. The web application may be alternatively referred to as theWebApp.

The user permissioning logic 6-7 may control access to the webapplication and prevent users from accessing information stored by theweb application that the users do not have permission to access. Theuser permissioning logic 6-7 may perform user tracking functions, changehistory tracking, and additional functions to allow users to access onlythe information they have permission to access.

The file format and data parsing logic 6-8 may be used to parse andinterpret data (e.g., media files, transcripts, etc.) stored in variousformats (e.g., TXT import, .MDB import, .PTF import, proprietary fileformats, etc.). The file format and data parsing logic 6-8 may parse andinterpret a synchronization index that controls text transcript displayin synchronization with other media being displayed.

The media delivery logic 6-9 may perform functions related to mediadelivery. For instance, the media delivery logic 6-9 may provide mediaencoding, HTML5 browser support, HTTP Live Streaming, and media (e.g.,iTunes) File Sharing.

The mobile computing device function logic 6-10 may perform functionsrelated to using functions of the mobile computing device in conjunctionwith the web application. For example, the mobile computing devicefunction logic 6-10 may allow the web application to leveragefunctionality of a smartphone (e.g., iPhone functionality) functionalityof a tablet computer (e.g., iPad), functionality of an Internettelevision device (e.g., Apple TV functionality), and functionality ofother operating system and mobile computing device technologies.

The search logic 6-11 may enable searching of text transcripts managedby the web application. For instance, the search logic 6-11 may enablesearching of all text transcripts, searching of all text transcripts fora particular case, searching of a text transcript for a particulardeposition, etc. The search logic 6-11 also may enable searching ofcorresponding options for alternative applications.

The annotation and edit logic 6-12 may enable adding and editingannotations of a text transcript. The annotation and edit logic 6-12 mayenable a user to add notes to a text transcript, lock editing of a texttranscript, link a counter designation to a direct designation, mergedesignations, bulk import designations, maintain confidentiality foredits to a text transcript, flatten designations, and add/editevidentiary ruling designations.

The video, display, and playlist logic 6-13 may control synchronoustext/video using a synchronization index. The video, display, andplaylist logic 6-13 may perform video clip sequencing, jump to locationfeatures, runtime calculation, display of font/appearance, display andcontrol of a scrub bar, timestamp editing, header/exhibits formatting,and variable speed playback control.

The linked exhibit logic 6-14 may enable display, linking, and controlof exhibits that are linked (or are desired to be linked) to atranscript. The linked exhibit logic 6-14 may include a PDF viewer andan image viewer to enable display of linked exhibits or exhibits a useris considering linking to a transcript.

The errata sheet logic 6-15 may enable a user to display, edit, andcontrol an electronic errata sheet for a text transcript. The erratasheet logic 6-15 may cause display of errata sheet entries with theoriginal text in the transcript. The errata sheet logic 6-15 may havedual-view edit features and view toggle features.

The output formatting logic 6-16 may control output of media and/or textin terms of display on a display screen or in a printed format. Theoutput formatting logic 6-16 may provide screen view control, 2-up viewcontrol, and print options (including PDF printing).

The scheduling and invoicing logic 6-17 performs scheduling andinvoicing features and may leverage mobile computing device features.The scheduling and invoicing logic 6-17 may provide calendarfunctionality, invoicing functionality, recurring paymentsfunctionality, scheduling functionality, subscription modelfunctionality, shipping functionality, and payment gatewayfunctionality.

The variable content logic 6-18 may control variable content display.The variable content logic 6-18 may control display of advertisements(e.g., advertisements offered by the company hosting the webapplication, location-based advertisement, commercials and PSAs, etc.),new music, movie trailers, and branded applications (e.g., corporatenews applications, educational applications, etc.).

Exemplary Synchronization Index

In some implementations, the function of a “synchronization index” is tolink a word or range of words with certain media (e.g., multimedia). A“synchronization index” is an electronic file that may take any ofseveral forms. For example, a synchronization index may be: 1) aspreadsheet with a time value corresponding to a word or range of words;2) a table with a time value corresponding to a word or range of words;3) a database, such as a Microsoft Access database, with a time valuecorresponding to a word or range of words; 4) an xml-type or html-typeformatted listing that associates a time value corresponding to a wordor range of words, for example as may be found in a SynchronizedMultimedia Integration Language (SMIL) formatted file or similar; 5) anyof the above, substituting the time value with a video frame count orany other marker or mechanism to mark a discrete portion of a media(e.g., multimedia) file a time t_(x) into the media (e.g., multimedia)between t=0 to t=end; or 6) a file, such as a portable document format(i.e., “PDF”).

The synchronization index itself may, then, include the transcript andthe timing values, or positional values, for the associated media (e.g.,multimedia). In this regard, for display of synchronous text and video,only the synchronization index and the associated media (e.g.,multimedia) may be needed, along with a system (e.g., one or morehardware devices operating software with logic) configured to utilizeeach.

The synchronization index, optionally, may include information relatingto the location and/or characteristics of the corresponding media (e.g.,multimedia), so the system can retrieve the multimedia for synchronousplayback. This may be a pathname/filename for retrieval of multimedia, aUniform Resource Locator (“URL”) specifying a location of themultimedia, a variant playlist, and so on.

The synchronization index also may include information relating topredetermined text display settings, for example defaults of text font,size, color formatting and so on selected to optimize an orderlydisplay.

Mobile Computing Devices

Techniques described in the present disclosure, or a portion thereof,are intended to function on a computing device, such as a mobilecomputing device. In the embodiments, a mobile device may comprise adisplay, a battery, a user interface, such as a keyboard, touchscreen,etc., and a memory, such as a flash memory or hard drive. In addition,the mobile device may comprise a wireless networking capability, such asa wireless Wi-Fi communications device, or a wired networkingconnection. The mobile computing device may utilize a known mobileoperating system, such as WebOS, Android, or IOS.

A mobile computing device generally comprises all or a subset of thefollowing features: a viewing screen; a battery supply for extendedperiods of use or standby without needing to be plugged in to anelectrical outlet; an operating system; an input mechanism, typically akeyboard rendered to a touch-sensitive viewing screen, or akeyboard/keypad that is integral to the device as opposed to a full sizekeyboard tethered or attached to the device by USB connection or othermeans; a networking mechanism that allows the device to communicate overa wireless network or telephone/data network to send and receive data(for example by Wi-Fi, 802.11a/b/g/n, Bluetooth, 3G, 4G, and so on); aquick start up time, relative to a longer start up times typicallyassociated with booting a computer running a full version of a MicrosoftWindows operating system; a utilization of solid state memory or flashmemory in preference to hard-drive memory; a quiet sound when operatingrelative to use of noisy fans and heat sinks found in desktop computers;and a physical size, weight, or form factor that would favorportability.

Common examples of a mobile computing device include, but are notlimited to, 1) a tablet computer, such as the device sold by Apple underthe iPad trademark, or others including HP Slate, Dell Streak, Asus EeeTablet, Compaq Tablet, Notion Ink Adam, MSI running Android, Quanta, ICDVega, Google Tablet, HTC, Archos 5 Internet Tablet, Asus T91mt Eee PC,Panasonic Toughbook H1 Field, Toshiba Portege M780-S7220, Samsung GalaxyTablet, and so on; 2) smartphones, including, but not limited to theApple iPhone (3G, 3Gs, 4), phones running the Android operating system,phones sold by Research in Motion sold under the trademark BlackBerry,Samsung Galaxy Smartphones, Nokia N8; and other smartphones; 3) handhelddevices, personal video recorders (PVRs), personal media players (PMPs),mobile gaming devices, personal digital assistants (PDAs), portablemusic devices (e.g., the Apple iPodTouch), and so on.

For simplicity throughout this disclosure, the term mobile computingdevice may be used interchangeably with the term tablet computer (e.g.,iPad), but it is not intended that the term mobile computing device belimited to a tablet computer (e.g., the Apple iPad). Any examples orimplementations utilizing a tablet computer (e.g., an Apple iPad) devicespecifically are illustrative only and are not intended to limit thescope of the disclosure. Certain illustrative examples may describe atablet computer (e.g., the iPad) as implementing techniques describedthroughout the disclosure, but the disclosure is not limited to a tabletcomputer (e.g., the iPad). The terms “web application” and “mobilecomputing device software” are described in greater detail, below.

Errata Sheet Creation

In the present disclosure, a user loads an electronic transcript to aweb-based software utility, referred to as the web application or the“WebApp.” The WebApp is configured to accept most industry standardtranscript formats. These may include: .txt, .ptx, .mdb, .html, .xml,and so on. Generally, these transcripts are numbered by page, and eachpage has an identical numbered series of lines that always correspond tothe same text to ensure a page:line cite is always static. In afinalized transcript, the page and line numbers will always correspondto the same testimony text from one copy of a transcript to another.This ensures a page:line citation by an attorney, or the court, willalways refer to the same location.

The web application re-formats the transcript into an xml-type format,generally described as associating a range of words with a specificpage:line reference, as would be the case if each line of testimony wasinserted into a series of database cells to comprise a column. AGraphical User Interface (GUI) allows a user to edit the text of anyline of testimony. The purpose of this edit is to notate an error intranscription that would need to be included in an errata sheet. Thesystem notes the original text of the line and compares it to thealtered text. In commercial word processing software, this process isoften referred to as “redlining.” New or added text can, at the user'sselection, be color coded to denote it as a change from the originaltext. Any combination of font characteristics can be selected by a userto denote that text that should be deleted, that text that should bealtered, or that text that should be added. Font characteristics mayinclude strikethrough, font color, font size, font, backgroundhighlighting, and similar characteristics.

The user can, by way of the GUI, instruct the system to run a report ofthe changes made and compile them in an errata sheet. The user can electto view either the original content of the original line of text or thealtered text of the revised line of text, or a markup with all changes.

A reviewer of the transcript can employ an additional step to confirmthe accuracy of the court reporter's transcription. Optionally, the textof the transcript can be synchronized to the video. In concert with asynchronization index, the user can click or perform a gesture upon aline of testimony to cause the video to begin to play back at thatportion of the video where those words were spoken. In this way, adeponent when reviewing the court reporter's written transcription ofhis testimony is not limited to his recollection of what was said, butrather has the exact video recording with which to compare the writtentranscription for accuracy. Moreover, attorneys can review the erratasubmission in conjunction with the actual video recording to ensureaccuracy of the transcript. With the written transcript and thewitness's recollection alone, without the video, there exists thetemptation for the witness to change his testimony to what he wished hewould have said at the deposition (for example, a “Yes” to a “No”),which is not the proper function of an errata submission, rather than toaccurately reflect through errata that which was actually said or tocorrect minor spelling, punctuation or typographical errors.Accordingly, the system may facilitate more timely and trustworthyerrata submissions, reduce the temptation for abuse because the video isavailable as a check, and ensure an accurate transcript displayselectronically in the courtroom as synchronized, scrolling text with avideo image, which may improve inefficiencies inherent in other systems.

Once the transcript has been reviewed by the witness in the system andchanges for accuracy have been made, the user can 1) create a writtenhardcopy report that aggregates all changes into an errata sheet, as hashistorically been the practice, for signature and affixation to theofficial hardcopy of the original transcript, and 2) transmit to theopposing party or the court the resultant electronic file, whichincludes the witness's changes.

Any of the software operators, preferably including those using mobilecomputing devices, including opposing parties, can 1) quicklyelectronically jump to succeeding or preceding errata notations (ornavigate via hyperlinks), 2) check the errata notations for accuracyagainst the video, 3) ensure the accuracy of the transcript as it willdisplay with the trial presentation software, and 4) play video in thecourtroom that contains information from the witness's errata sheetsubmission as though it was part of the court reporter's transcription.

Advantageously, an errata sheet prepared and communicated in this manneralso may be electronically tracked, for example by affixing anunalterable, encrypted time and date stamp from a trusted, independentsource, should the submitter ever need to verify that the erratacorrections were completed within the time contemplated by the governingrules of procedure and communicated to opposing counsel or the court.

FIG. 7 illustrates an example interface that may be used in handlingerrata sheet operations. As shown, in FIG. 7, a user may view adeposition video and a deposition transcript in a synchronized manner,and manipulate an errata display area to make a change to a depositiontranscript to be added as a change in an errata sheet. In the exampleshown in FIG. 7, a user “Deponent” selected line 15 of the depositiontranscript and changed the testimony “Yes” to “Yes, I believe so.” Asshown, the errata display area shows the change made, the user who madethe change, and the date on which the change was made. In addition, thedeposition transcript has been updated to indicate the change made tothe transcript at line 15 such that the original transcript and erratachanges may be displayed together in synchronization with the depositionvideo. In some examples, the changes entered in the errata display areamay be aggregated into an electronic errata sheet associated with thedeposition transcript.

Merging and Flattening of Designations

Law firms are often tasked with transcript management on behalf of theirclients. They commonly utilize a computer spreadsheet application, suchas Microsoft Excel, to track changes, and they use a word processor,such as Microsoft Word, to type out their designations, sometimes intabular format or longhand. Use of spreadsheets and word processors maybe cumbersome and inefficient for transcript management.

A common way a deposition is cited is in the longhand, exemplified as“Page 10, line 2 to page 10, line 7.” A shorthand for such a designationunderstood by trial presentation software would be, for example,“StartPage.StartLine+EndPage.EndLine.” In the example, the shorthandwould be 10.2+10.7.

Attorneys conduct review of transcripts in the context of internalreview, and then in preparing for trial. In conducting internal review,or in transitioning from internal review to trial, it is often desirableor necessary to flatten certain designations. For example, onetranscript may be reviewed by Plaintiff's counsel, and it may be issuecoded to comprise a certain number of “page:line to page:line”designations that are associated to a particular issue. For example, oneissue code may be titled “liability” and another “damages.” Using thesystem described throughout this disclosure, a user may combine, orflatten, two or more issue codes into a common issue. In this example,the user may elect to flatten the “liability” coded designations withthe “damages” coded designations into one common issue called, say,“Plaintiff's direct designations.”

Similarly, one designation may be created and another one may be addedat a later time but with a range that abuts the first designation. Ifthese abutting designations have the same issue code, for example, theyshould be able to be merged into one designation. The disclosed system(e.g., the web application) comprises logic, such as executable programcode, configured to automatically merge, at the user's choosing,abutting designations with the same issue code.

A list of top level, global, or roll-up issue codes may includePlaintiff's direct designations, Defendant's counter designations,Plaintiff's counter-counter designations, Defendant's directdesignations, Plaintiff's counter designations, Defendant'scounter-counter designations, Plaintiff's objections, and Defendant'sobjections, as well as Confidential, Restricted Confidential, andAttorneys' Eyes Only. As in the example above, a user may select thetranscript sections issue coded “liability” and “damages” and specifythat they be combined into another single issue code such as“Plaintiff's direct designations.”

In reference to the desirability of flattening designations prior todisplay of video clips in court, it may be advantageous for an operatorof trial presentation software to flatten designations before creatingvideo clips, because this reduces the total number of video clips tocreate as well as the number of edit points, thereby facilitating moreefficient workflow. For example, if a plaintiff designates for deponent“John Doe” testimony in the range 10.1+10.5 and 10.9+10.25, and ifdefendant counter-designates 10.6+10.8, the one video clip created maybe for 10.1+10.25. More typically, and problematically, threedesignations would have likely resulted in three distinct video clipsand they would be played as a multiclip, or series of individual clips:

Page: line range Clip title 10.1 + 10.5 JD01001 10.6 + 10.8JD01006-counter 10.9 + 10.25 JD01009

Using the system described throughout this disclosure, the logic of thesoftware of the web application may recognize adjoining ranges ofdesignated testimony. The logic may be employed to join, or flatten,these adjoining ranges. Further, rules may be employed to flatten onlythe desired issue codes, preserving the original range specification of“page:line to page:line” and issue tag should the user elect to undo aflattening operation, or should the user need to perform a calculationof video run duration, described below.

FIG. 8 illustrates an example of merging designations. As shown, aplaintiff has made initial designations to a deposition transcript and adefendant has made counter designations to the deposition transcript. Inthis example, the system recognizes that the plaintiff designations andthe defendant designations represent adjoining ranges of designatedtestimony and, based on the recognition, merges the plaintiffdesignations and the defendant designations. The system displays amerged designations area that shows the designations that have beenmerged adjacent to the deposition transcript, which still shows theplaintiff designations and the defendant designations. Using the mergeddesignations area, a user may verify whether or not the merging wasproper and, if so, create a single video file that corresponds to themerged designations.

FIG. 9 illustrates example data structures that may be used in mergingdesignations. As shown, a first user has designated page 117, line 13 topage 130, line 16 and a second user has designated page 121, line 0 topage 135, line 10. The system compares the position information (e.g.,page and line numbers) of the designations made by the first user withthe position information (e.g., page and line numbers) of thedesignations made by the second user and determines that thedesignations overlap based on the comparison. In response to thedetermination that the designations overlap, the system merges thedesignations into a single, merged data structure. The merged datastructure includes all of the designations made by the first user andthe second user, encompassing the earliest starting point fordesignations to the latest ending point for designations. For instance,as shown, the merged data structure has position information indicatingdesignations from page 117, line 13 to page 135, line 10.

FIG. 10 illustrates an example interface for assigning issues todesignations. As shown in FIG. 10, a user may select one or more rangesof designations and assign one or more issues to each selected range ofdesignations. The interface enables a user to add additional issues andapply all issues to a particular designation. The interface also allowsa user to select a color with which the designations tagged with aparticular issue will be displayed on a deposition transcript. Theissues assigned to designations may be used for flattening and mergingdesignations, as described above.

Locking Designations

In communicating designations between parties, the system prevents oneparty from designating testimony that has already been designated by theother. For example, if plaintiff designates 10.1+13.22, then thedefendant should be prevented from counter-designating testimony in thesame range. One way of understanding this function is with reference toFIGS. 11A-C, where each party has its own work pane display of thetranscript. In this manner, Plaintiff uses the left-most work pane tomake annotations or to perform issue coding. The Plaintiff thencommunicates to Defendant that its designations are complete, and theDefendant begins to make counter-designations in the right-most workpane. Navigation of the transcript can operate on both work panessimultaneously, so it becomes easy for a Defendant to determine ifcounter-designations are needed in certain areas, without having toworry about overlapping annotation colors. The logic of the systemprevents overlapping designations between parties. If an overlappingpage:line is selected, the software snaps Defendant's selection to theavailable lines and/or provides an error message. As above wherePlaintiff designated 10.1+13.22, if defendant inadvertently selects13.15+14.10, the designation would automatically correct to 13.23+14.10,disregarding the overlapping section. The result may offer a clean,concise view of designations within the two work panes.

Where the parties make a simultaneous designation, for example bysubmitting direct designations to one another at a mutual time deadlineset by a court, those page:line ranges can be merged into a distinctissue code, such as “both parties,” or “agreed upon designations,” andbe locked to prevent objection by either party.

Linking Designations

The web application may comprise logic configured to link onedesignation with another designation. For example, in a depositiontranscript a Plaintiff may proffer a set of direct designations for awitness. Thereafter, the Defendant may desire to add material forcompleteness in a Defendant's counter-designation should the Defendantbelieve Plaintiff's direct designations were not a complete or accuraterepresentation of that portion of the testimony. In practice, however aPlaintiff may elect to drop its initial designation, making it alaborious process for the Defendant to eliminate corresponding Defendantcounter-designations that should be removed as a result. Accordingly,the logic configured to link one designation with another designationprovides functionality to automatically cause the deletion of acounter-designation if the designation to which it is linked is firstdeleted.

This logic may be further described with reference to FIG. 11A-C, whichillustrates a Plaintiff's work pane and a Defendant's work pane. Asshown, a first designation is made by Plaintiff in Plaintiff's workpane. The Defendant's counter-designation is made in the Defendant'swork pane, with logic to allow, or to compel, Defendant to identify thatportion in Plaintiff's work pane that would correspond to Defendant'scounter-designation. Defendant's counter-designation may be linked tomore than one of Plaintiff's direct designations, with the linkeddesignations being listed in Defendant's work pane. Should each ofPlaintiff's designations within that list be removed, the logic causesremoval of the corresponding counter-designation.

The Defendant may link each counter-designation to at least onePlaintiff designation using a graphical element, for example byselecting a range of text in Defendant's work pane and then dragging a“lasso” into Plaintiff's work pane to identify a page:line range toindicate the Plaintiff's selected testimony to which Defendant is tyinghis counter-designation. Again, one Defendant counter-designation may betied to more than one of Plaintiff's designations. If all of those ties,or linkages, are severed by Plaintiff withdrawing his designations, thenthe logic may be configured such that Defendant's counter-designationsare removed, or are identified as counter-designations that ought to beremoved.

Alternatively, if Defendant has linked a counter designation to aPlaintiff's designation, and if the Plaintiff subsequently modifies therange of that designation, the Defendant can be notified, by the logicof the system, that one designation to which a Defendant'scounter-designation has been linked has been modified. The Defendant canmake a determination as to whether his counter-designation should bedeleted entirely, deleted in part, or left alone. With this logic tolink designations, the parties are provided a new tool that may makeefficient a work process that has historically been very laborious andfraught with potential for error. FIGS. 11B and 11C illustrate examplesof linking designations according to one embodiment.

Batch Import of Designations

It may be advantageous to use the web application to import more thanone transcript designation at a time. For example, when parties tolitigation exchange their designations, each party needs to be able toimport the designations of the other into his own working copy of thetranscript. As another example, video clips may have been created withintrial presentation software that need to be integrated into a transcriptannotation utility. As part of the logic of the web application, a usermay batch import more than one designation at a time. This can beaccomplished by creating a loadfile with a predetermined format tocommunicate the batch of designations. For example, a loadfile using theStartPage.StartLine+EndPage.EndLine entries in series may be used. Thisformat is the same as may be used to batch create video clips withintrial presentation software applications. Upon ingest of the batch, theuser may be presented a graphical user interface to assign an issue toeach of the designations. FIG. 10 illustrates an exemplary interfacethat can be used to batch import more than one designation and assign anissue code to one or more designations.

Printing

Once parties to litigation have exchanged deposition designations, andobjections thereto, with one another, it is often a requirement that aprinted hardcopy be prepared for submission to the court. The printedhardcopy is also useful for the attorneys' own reference.

These submissions are typically color-coded copies of depositiontranscripts for the convenience of the court and court staff.Historically, these color copies have been laboriously prepared by handusing a colored high-lighting marker on previously printed copies of thedepositions. This method is extremely time consuming, it creates greatpotential for human error, and the work product is not easily revised orcombined with the submission prepared by opposing legal counsel.

Moreover, during the deposition designation and counter-designationprocess, each party may also lodge objections to any number ofdesignations or counter-designations for the purpose of asking the courtto exclude the testimony from introduction into evidence under the rulesof evidence. An objection is a written note to correspond to anannotation, that being a selected page:line range of the transcript thatis commonly assigned an “issue code” meaningful to the user. The writtennote may comprise a reference to the evidentiary rule being relied uponby a party to warrant exclusion of the testimony by the judge as well asa short description or argument on the matter. It has been verydifficult to view the objection and the testimony to which the objectionwas made, without using at least two separate documents or a table froma word processing application (e.g. Microsoft Word) That is timeconsuming to prepare and has a high probability of human error.

In the described system, the user is provided the facility to printhardcopies of the deposition transcript, including highlighting for eachissue code in a different color, or a different font. For example, theuser may be able to print objections on the same page as the transcriptand in a location that is adjacent, or substantially adjacent, to thetestimony itself. The logic in the software of the web application,preferably working in concert with software operating on a mobilecomputing device (including optionally utilizing the mobile computingdevice to communicate the print job to the printer), may be able to 1)print in a portrait orientation with two transcript pages comprising acolumn on the left half of the printed page, leaving the right half ofthe page for notes, such as an objection, to appear adjacent to thetestimony to which a party has lodged an objection, and 2) print in alandscape orientation with a transcript pages comprising a column on theleft half of the printed page, leaving the right half of the page fornotes to appear adjacent to the testimony to which a party has lodged anobjection. Printing in this manner allows one convenient document to beproduced where a judge can evaluate an evidentiary objection whilesimultaneously accessing the testimony to which the objection applies.The reader also can examine the context of the overall transcriptbecause the transcript contains highlighted designations as well as thenon-designated portions, comprising the complete transcript.

A deposition designation has a color associated with it so the readercan immediately understand which party proffered the submission,including by reference to a color key that may be printed on the samepage. Likewise, a graphic element, for example an elongated, coloredbracket or a rectangle bounding the designated text, may be used toidentify the party making an objection and the range of text to whichthe objection applies.

Printing may encompass communication with a printer to print on paper,creating a file in .pdf or .tif format, or other similar format,suitable for transmitting by email or printing at a location remote fromthe user, or rendering a file type for use with a mobile computingdevice, such as an electronic book format or similar (including .txt,.html, .ps, .pdf, .djvu, .epub, .fb2, .azw, .pdb, .lrf, .lrx, .wol,.tr2, .tr3, .aeh, .lit, .exe, and .rgo).

This printing method provides a new convenience to the reader, as atleast two documents—a highlighted transcript and a table of objectionsand page:line cites to which objections are lodged—are consolidated intoone. This printing method may solve a longstanding logistical problemfor litigation professionals needing to submit deposition designationsand objections to a court.

FIG. 12 illustrates an example printed page output in a portrait view.FIGS. 13A-B illustrate an example printed page output showing onlydesignated portions of a transcript. FIG. 14 illustrates an exampleprinted page output in a landscape view. FIG. 15 illustrates an exampleof minuscript printing showing highlighted designations.

Electronic View of Printed Format

In addition to the printed copy, it may be advantageous to replicate ona computer screen the convenient layout of having objections adjacent tothe testimony to which objections were lodged.

Additionally, the judge can be provided facility to make an evidentiaryruling within the system. With reference to FIG. 16, the court isprovided the facility to make evidentiary rulings on objections, hereexemplified in the right-most column by an affirmed/overruled toggle inthe form of a radio button. As a result, if the court elects to make aruling, the date and time of the ruling is stored by the software, andsuch a ruling can be immediately communicated to the parties. Theparties can access the electronic document to learn what testimony hasbeen excluded and what has been permitted, and a report can be generatedand printed or saved for offline study and evaluation.

Transcript Formatting

An example format for the synchronization index for use with thedescribed system is an xml-type format, an example of which is shown inFIG. 17. This xml-type format comprises tags to denote a transcriptheader, a transcript body, and a transcript footer. Metatags may also bepresent for display, including timed display, of graphics, images,advertising, secondary multimedia other than that primarily associatedwith the text of the synchronization index, logos and settings therefor,text, including foreign language translations, sounds, and so on.

Transcript annotations can be saved as an integral part of the xml-typeformatted synchronization format. The example shown in FIG. 17 isintended to be illustrative of the xml-type format only and is notintended to represent an actual, working synchronization index with allinformation included in the synchronization index. Actualimplementations may include more, perhaps much more, information.

Run Time Calculation

When video clips are created in trial presentation software, a reportquery can be executed to calculate and output the run time duration ofeach clip and the designated text. The report may be called a durationlist, or “durlist.” For the reasons mentioned above, litigants need thisinformation, in part to determine whether they should add or removetestimony.

Historically video clips were created within the trial presentationsoftware to generate a duration list. Because trial presentationsoftware is often a tool not used by an attorney but rather a trialconsultant with special expertise, typically employed only at a timevery close to trial, it would be useful if the attorney could create aduration list apart from the trial presentation software and within anapplication used to annotate transcripts. In some examples, all of thiscan be accomplished on a mobile computing device.

As an example of cumbersome and inefficient workflow often experiencedby litigators, the attorney creates designations using a highlighter ona printed transcript; support staff enters those designations into atranscript management utility, which carries the potential for error;the transcript management utility case is sent electronically to thetrial consultant; the trial consultant runs a query from the transcriptmanagement utility to extract a .csv formatted file corresponding to theissues designated; the .csv is re-formatted to render a text file inStartPage.StartLine+EndPage.EndLine format to be used as a loadfile forvideo clip import to a trial presentation utility; the trialpresentation utility is opened, and the .txt loadfile is used to importdesignations; the clips within the trial presentation utility arequality reviewed to ensure load errors have not occurred, for example aswould exist if a designation began or ended on a line with no timestampwithin the trial presentation utility .mdb; the video clips arefine-tuned at the beginning and end of each clip; the video clips arerenamed to sort properly in page order within a duration list, forexample by padding a “0” for pages 1-99 or by adding a prefix or suffixto identify the designator; the “durlist” command is executed, requiringthe trial consultant to select the witness having video clips loaded andthen manually parse out only the subset of video clips for which aduration list is sought from among the entirety of video clips presentfor a witness (e.g. clips within the trial presentation utility areidentifiable only by clip name and do not carry any issue codeddifferentiation that may have been present in the transcript managementutility); the duration list report is created from the clips identifiedby the trial consultant and printed, or saved electronically, foranother quality review against the transcript management utility case;and then the duration list is forwarded back to the legal assistant orattorney.

With the described system, logic is employed in the software of the webapplication, and preferably within software executed on a mobilecomputing device, to calculate a duration list, without necessity ofcreating discrete video clips, by performing calculations from thesynchronization index itself. For example, if a user annotates apage:line range within a transcript, the software examines thesynchronization index to compute the time difference between videocorresponding to EndPage.EndLine and StartPage.StartLine. In thismanner, computations can be output for each annotation, for allannotations that comprise one issue, for merged issues, for more thanone issue, and for the totality of all annotations.

In this manner, the attorney can very quickly determine theappropriateness of content being designated because he has immediateknowledge of the run duration of the annotations being made. Thetraditionally cumbersome and inefficient workflow is greatly improved.This provides the attorney a heretofore unavailable advantage, in thatthe strategy of determining what video to play in the courtroom has beena strategy historically left until the last moments of trialpreparation, often as late as the day before video is to be played inthe courtroom. This trial preparation strategy can now be deployed muchearlier in the litigation process, leaving available more preparationtime for the attorney in the moments immediately before trial, or in theevenings on trial days when preparation time is most scarce. Work thathad consumed the attorney's time can be shifted to a trial consultant,as the actual play duration of video clips may vary slightly from thecalculated duration list because the trial consultant can often finetune the video to eliminate long pauses, false starts and stops, andcorrect inaccuracies or imperfections in timestamping of thesynchronization index. With the described system, this fine-tuning,however, is no longer prerequisite to the attorney's evaluation ofoverall run durations historically generated once video clips had beencreated, historically a very time consuming task at trial.

FIGS. 7, 11, and 16 each show a displayed output of run timecalculation. As shown, the system computes and displays a total run timefor all designations, a total run time for plaintiff designations, and atotal run time for defendant designations. The system may compute thetotals in real-time as a user is viewing the transcript and video of thedeposition, and editing designations for the transcript. The user mayview the run time calculations and quickly perceive how changes beingmade to the designations impact run time of the corresponding video. Insome implementations, the system may continuously monitor changes to thedesignations and, when a change is detected, the system may referencethe synchronization index to extract timing data associated with the newdesignations and compute the run time totals for the new designationsbased on the extracted timing data.

Synchronized Exhibits

When an attorney reviews a videotaped deposition and its transcriptafter the deposition, it is often helpful to have access to depositionexhibits that may have been physically before the witness during a lineof questioning. It is a feature of the present disclosure to providewithin the synchronization index information that would provide for thedisplay of a document at the correct time within the deposition. Forexample, when an attorney places a document before the witness, marks itas a deposition exhibit, and asks the witness to identify it, metadatawithin the synchronization index can be configured to display that firstpage of the exhibit at that point within the deposition video. Thatpoint in time also corresponds to a page:line cite within the depositiontranscript. If, in the attorney's next question he draws the witness'sattention to the third page of the deposition exhibit, metadata withinthe synchronization index can be configured to display that third page,and so on.

Accordingly the synchronization index contains information for selectingthe image to display, for the time at which the image is to appear, oran “in” time marker, and for the time at which the image is to beremoved, or an “out” time marker. A series of such instructions within asynchronization index is generally referred to as “synchronizedexhibits.” It may be advantageous to allow data for synchronizedexhibits to be independent of video clips and “in” and “out” edit pointsfor video clips. Such independence allows video clips to be edited,merged and flattened without necessity to re-link the correct exhibitthat should be simultaneously viewed with each video clip. Existingtranscript management and trial presentation utilities lack thisfacility, making their use cumbersome and inefficient.

FIG. 18 illustrates an example of a linked exhibit. As shown, theinterface in FIG. 18 includes a linked exhibit area where a user canview and add/delete/edit linked exhibits to the transcript displayed. Inthe example shown, an exhibit has been linked to the designations madeto the displayed transcript. The exhibit is displayed adjacent to thetranscript testimony to which the exhibit is linked. The linked exhibitmay be displayed at a time when a portion of the deposition videocorresponding to the designations is displayed. Users may be able tointeract with the linked exhibit area to add additional linked exhibits,edit existing linked exhibits (e.g., move the linked exhibit to adifferent portion of the transcript), or remove linked exhibits.

The Web-Based Software Application

In some examples, the web-based software application may be anapplication coded within the Ruby on Rails (“Rails” or “RoR”) opensource web application framework for the Ruby programming language.

The web application is used as a case management system to facilitatemanagement of users, user permissions, team collaboration, transcriptmanipulation, and distribution of synchronization indexes and multimediaretrieval instructions for a mobile computing device. Referring to FIG.6, the web application may comprise user permissioning logic 6-7; fileformat and data parsing logic 6-8; multimedia delivery logic 6-9; mobilecomputing device function logic 6-10; search logic 6-11; annotation andedit logic 6-12; video, display, and playlist logic 6-13; linked exhibitlogic 6-14; errata sheet logic 6-15; output formatting logic 6-16;scheduling and invoicing logic 6-17; and variable content logic 6-18.The web application may include software code intended to function as a“cloud computing” application, software running on a server remote fromusers who access it and its functionality by way of an internetconnection, an internet or web browser, and a computer or, possibly, bya mobile computing device. The software code may be stored in anon-transitory computer-readable storage medium and may be executed byone or more processors to perform operations described throughout thisdisclosure.

First, user permissioning logic 6-7 is present, generally to ensure eachuser has an account and that the account holder have access only to thatcontent he has been authorized to receive. The user permissioning logic6-7 may be understood with reference to a series of use cases. Forexample, consider a situation in which a plaintiff's attorney, adefendant's attorney, and a judge all are using the web application toreview a deposition transcript. In this example, the plaintiff'sattorney, the defendant's attorney, and the judge should be able to viewsome information that is public and common to all users, but should notbe able to view information that is private or confidential to one ofthe other users. The user permissioning logic 6-7 ensures that theplaintiff's attorney, the defendant's attorney, and the judge see onlythe information that they are permitted to view. For instance, the userpermissioning logic 6-7 controls display to the defendant's attorney andthe judge, so that they can perceive designations shared by theplaintiff's attorney, but not confidential notes or annotations to thetranscript made by the plaintiff's attorney. Similarly, the userpermissioning logic 6-7 controls display to the plaintiff's attorney andthe judge, so that they can perceive counter designations shared by thedefendant's attorney, but not confidential notes or annotations to thetranscript made by the defendant's attorney

Second, file format and data parsing logic 6-8 is required. This logicis used to ingest transcripts from a user. Within the legal industry,for example, there are a number of various electronic formats fortranscripts, including .txt; .ptf, a file format most often associatedwith transcripts capable of being read by, or exported by, thecommercially available transcript utility called LiveNote byThompsonReuters; .ptx (or occasionally as an .exe), most oftenassociated with transcripts capable of being read by, or exported by,the commercially available transcript utility called “eTran” byRealLegal; .mdb, the Microsoft Access formatted transcript type fromVerdict Systems and its software called Sanction II; .cms (or .mdb,Microsoft Access) formatted transcript type from in Data and itssoftware called TrialDirector; and a number of other file formats usedby common transcript management and/or trial presentation utilities.

The purpose of the file format and data parsing logic 6-8 is to ingest atranscript, to parse it according to its known formatting attributes forthe purpose of extracting information including the text transcription,page and line numbers, headers and footers, multimedia associations,document exhibit listings and document exhibits, and timestamp data (orvideo frame data) and rendering all of said information into anintermediate, standardized format (herein referred to as the “xml-type”format) for use in the described system as a synchronization index thatcan be communicated to, and understood by, a mobile computing device.

Third, multimedia delivery logic 6-9 is provided. The multimediadelivery logic 3-9 functions to deliver multimedia content to a user.For instance, the media delivery logic 6-9 may provide media encoding,HTML5 browser support, HTTP Live Streaming, and media (e.g., iTunes)File Sharing.

Fourth, mobile computing device function logic 6-10 is explained. Mobilecomputing devices may be different from one another, so a separatemobile computing device function logic 6-10 is often needed tofacilitate communication of data and multimedia to the device. Mobilecomputing device function logic 6-10 may include logic for the AppleiPhone (running the Apple operating system called iOS, including iOS 3.2and iOS 4.0-4.2), the Apple iPad (running iOS, including iOS 3.2 and iOS4.0-4.2), mobile computing devices running an Android operating system,and Research in Motion-type or Blackberry-type mobile computing devices(for example those utilizing Blackberry operating systems), and mobilecomputing devices utilizing the Symbian3 environment (to include devicesmanufactured by Nokia, such as the smartphone device sold with thetradename “N8”).

By way of example, the mobile computing device function logic for theApple iPad is written in Xcode for iOS4.2 and is compiled to communicatewith the web application. A user, having the correct permissions, canaccess the web application for retrieval of one or more synchronizationindexes and corresponding multimedia. Fifth, search logic 6-11 ispresent to facilitate search of a synchronization index or a collectionof multiple synchronization indexes. Once a user has used a mobilecomputing device to select a synchronization index, that synchronizationindex can be communicated to the device in whole and cached there, or itcan be communicated to the device in part and subsequent portions of thesynchronization index can be communicated to the mobile computing deviceas the user requires. Since many mobile computing devices have limitedmemory storage capacity and RAM, relative to laptop computers, desktopcomputers and servers, it may be advantageous to allocate certain tasksto that device best equipped to handle the task, while factoring inspeed, efficiency, system resources and connectivity. A search for textis one such task, and that is one reason for search logic.

For example, if a user elects to retrieve the synchronization index fora single deposition having, say, a 300 page length if printed, themobile computing device may be able to download the entiresynchronization index and store it in memory. A text search of thatdeposition may reference the locally stored copy of the synchronizationindex and return a search result. This may be done with speed andefficiency on the mobile computing device.

If, however, the user expands the text search to include all depositionswithin that legal case of, say, 50 depositions rather than justsearching one 300-page deposition, the search controller provides logicto efficiently perform the search and return search results to themobile computing device. For instance, the search may first referencethe locally stored content, then the logic comprising the searchcontroller may ask the remote server to conduct the search on theremaining transcripts to be searched. Without necessarily communicatingthe entirety of all fifty synchronization indexes to the mobilecomputing device, the server can return to the mobile computing deviceonly the search matches. When the user selects one of those searchmatches, only then would the server deliver a larger portion of thatspecific synchronization index, or the entire synchronization index, tothe mobile computing device.

The search logic also facilitates search across vast libraries ofsynchronization indexes. It is contemplated that searches may involve,for example, searching lyrics of thousands of songs that may be hundredsor thousands of gigabytes in size. Such a search is best conducted on aserver. As another example, a user of an Apple iPod with 32 Gb of memorymay have a portion of that space allocated to, say, storage of 5000songs, 2500 of which have lyrics that have been downloaded and stored tothe device. A search query would first be executed upon the lyricsresident on the device and search results returned and displayed, and,at the user's option, the query would be extended to a server-sidesearch to encompass lyrics not resident on the mobile computing device,to return and display search results.

Next, annotation and edit logic 6-12 is provided to allow manipulationof a transcript. In general, this is the logic employed to facilitatethe transcript management and manipulation functions described above,including functions relating to creating annotations, notes, issuecodes, merging designations, flattening designations, lockingdesignations, and so on. Annotation and edit logic 6-12 communicatesupdates made on a mobile computing device to the remote server forstorage and sharing with other users.

Seventh, video, display, and playlist logic 6-13 is employed. The video,display, and playlist logic 6-13 may perform video clip sequencing, jumpto location features, runtime calculation, display of font/appearance,display and control of a scrub bar, timestamp editing, header/exhibitsformatting, and variable speed playback control.

Eighth, linked exhibit logic 6-14 may be used. As described above withrespect to synchronized exhibits, within a deposition, documents areoften placed before the witness to elicit testimony. These are calleddeposition exhibits. When reviewing the transcript, it is often helpfulfor the reader to have convenient access to deposition exhibits,particularly accessible as a document linked to the deposition textwhere the exhibit was utilized. Linked exhibit logic 6-14 may assist indelivery of material other than the transcript, by using thesynchronization index to retrieve that material. For example, the linkedexhibit logic 6-14 may comprise a hyperlink within the text displayed inthe transcript that a user may select for display of a particulardeposition exhibit. If, for instance, an exhibit is referred to as“Smith Exhibit 1” in the transcript, the logic of the linked exhibitcontroller would link that text to the document comprising Smith Exhibit1, perhaps a pdf image or a tiff image. A user following that hyperlinkwould be brought to the appropriate image.

Alternatively, for use with the Apple iPad, the linked exhibit logic maycomprise logic to display material without input from the user. Forexample, if the synchronization index is constructed properly, adocument can be linked for display at a time within the multimediawithout prior user input. While the user watches the multimedia, andsynchronous text display, a deposition exhibit may appear on the screenat the time it is relevant and would be useful for the user. The usercan elect to play/pause the multimedia, navigate to selected pages fromthe deposition exhibit, or remove the deposition exhibit from thedisplay and resume playback of the multimedia and text.

Ninth, errata sheet logic 6-15 may be used. The errata sheet logiccomprises logic to display only an accurate transcription of the media.A more specific description of the function of errata sheet logic 6-15is discussed, above, as is the way the logic operates to facilitatedisplay of an accurate transcript.

Tenth, output formatting logic 6-16 is present. This is the logicrequired to provide the various printed outputs, described above, forthe organized printing of a transcript with annotations and notes,including notes that can be printed substantially adjacent to the textto which it is associated.

Eleventh, scheduling and invoicing logic 6-17 is provided to facilitatethe ability of a user to see a calendar (e.g. for depositions) and topay for service. The web application may function as asubscription-based case management program with a recurring paymentcomponent, and this logic operates to facilitate processing of paymentsfrom a user account based on a subscription agreement.

Twelfth, variable content logic 6-18 is provided to make the systemadaptable for uses in various alternative industries. This logicgenerally relates to the graphical user interface and graphical elementsthat may be suited for a particular use or industry. This includes thedelivery of targeted advertising, branding elements, graphical “skins”to achieve a desired look-and-feel for a given application, commercials,and so on.

The web-based software application employs a number of controllers forvarious related devices, functions, and software associations.

In general, some function controllers for the web-based softwareapplication are used for operator manipulation of data via the mobilecomputing devices and have logic configured to perform a plurality ofactions, specifically: to upload, manage and run file/format andvideo/text data; search engine functionality; annotation and editingcapability; calculation of video run time from designations withoutfirst creating a video clip; adding and merging or flattening ofdesignations; cross-party communication to finalize designations;electronic errata sheet creation and updating; updating of issue codingand batch import of issue codes; a utility for printing with objectionsadjacent to designations; a utility for communication between parties;and a mechanism for a judge to participate by making evidentiary rulingsin the transcript management utility.

As shown in FIG. 6, a hosting server is used in conjunction with amultimedia server. A dual server system is shown for the present systemdue to the complex interactions between the web application, themultimedia server, and the plurality of mobile computing devices, aswell as the substantial volume of information being uploaded, managedand served to users.

As an example, the hosting server is a cloud based computing platform,such as the hosting service provided by Heroku. The Heroku Server ispreferably a platform as service/ruby on rails type cloud platformhaving open-source extensibility. The hosting server may deploy AmazonSimple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud(Amazon EC2). This server stores a SSL-encrypted case managementweb-based software application, all user information, and case datainformation, to optionally include synchronization indexes. The hostingserver is also the preferred location for the web application to beexecuted. A Heroku Server or other equivalent hosting server may controlthe cloud computing and data storage objectives described above withoutdeparting from the scope of the present disclosure.

An appropriate multimedia server is a server, or multiplicity ofservers, configured to deliver multimedia, preferably multimedia that isappropriate for the receiving device and the bandwidth available to it.Exemplary multimedia servers and related services to deliver multimediawith a number of variant playlists include a Brightcove HTTP streamingserver, or an Akamai HTTP streaming server, or a Wowza media server. Apreferred HTTP streaming server has open-source extensibility and isadapted for streaming many multimedia formats, including in variantplaylists, including audio, H.264, MPEG4, M4V type media files; HTTPLive Streaming (HTML5); MPEG-2 transport stream (H.264 video/AAC audio);and multimedia segmented for delivery on an Apple iPhone or iPad.Preferably, all multimedia data and text data is stored on themultimedia server.

The multimedia server may be configured to deliver streaming multimediato a mobile computing device directly. It may also deliver streamingmultimedia to a computing device directly, for example to a user of theweb application using a web browser. The multimedia server may also beconfigured to deliver multimedia to an intermediate computing devicethat may then transfer multimedia to a mobile computing device. Forexample, a user that desires access to multimedia using a mobilecomputing device may elect to store the multimedia on the mobilecomputing device itself. Apple's iTunes, for instance, may be used tocommunicate multimedia from a computer to an iPad or iPhone temporarilytethered to the computer.

The hosting server and the multimedia server are in electronic datacommunication with one another and with a plurality of mobile computingdevices operated by a plurality of users or operators, typically,attorneys and paralegals or commercial multimedia consumers. There aremany suitable mobile computing devices, as listed above. The iPad withiOS 3.2-iOS 4.2, and the iPhone with iOS 4.0-iOS 4.2 may be used. TheiPad tablet-type mobile computing device offers functionality, includingthe ability to multi-task in iOS4.2.

Specifically, the iPad is adapted to have a case/deposition reader withdeposition viewer and supporting issue code identification; annotationfunctionality; case and deponent level search-ability; HTTP LiveStreaming capability; annotation and time stamp editing; CoreText-typeengine for text display ability; and exhibit linking and display viaMPTimedMetadata with iOS 4.2. The iPhone is suitable for a subset of theabove functionality, primarily due to the fact that its screen issmaller than the iPad, and is therefore a viable choice for a mobilecomputing device adaptable to the present system. Specifically theiPhone is capable of case/deposition reading with deposition viewer andsupporting issue code identification; annotation functionality; and caseand deponent level search-ability. Any combination of the abovedescribed mobile computing devices may make case management andperforming the above manipulations now a nimble, in real time, taskinstead of the arduous and cumbersome modes of case management as areknown.

In some implementations, a video display device may be associated withone or both servers. The mobile computing devices may be in electronicdata communication with both the video display device and the servers.When a user performs a manipulation on a mobile computing device, suchas scrubbing to a certain point in a video, the output or display may beshown on the video display device, either alone or along with beingdisplayed on the mobile computing device. In this regard, the mobilecomputing device may be used to control display on a separate videodevice (e.g., a video display device in a courtroom, a video displaydevice in a conference room with several participants, a video displaydevice in a remote location, etc.).

In some examples, the data produced by the web-based softwareapplication is viewable on a plurality of mobile computing devices. Apredetermined gesture on a hyper-linked line may initiate identificationof desired video, may identify the elapsed time within that video, fetchand begin streaming the video at that time back to the mobile device,and finally begin scrolling the transcript in synchronization withremainder of the video as it plays. In these examples, a plurality ofmobile computing devices may be used by multiple users, and playback maybe provided via an independent display device associated with the remoteserver where data is stored. The web-based software application may havefull search functionality for data accessible to the web-based softwareapplication. The web-based software application also may allow users tolog identity (e.g., name, password, firm, mobile device serial number,phone number, etc.) and keeps record of multimedia files accessed, theportion of the transcript that would correspond to those segmentsviewed, and the individual who made/approved the designation.

Mobile Computing Device Software

A mobile computing device runs software for transcript management.First, the mobile computing device software has logic configured tocommunicate with the web application running on the hosting server. Thiscommunication allows transfer of a synchronization index to the mobilecomputing device. As described above, the web application ingeststranscripts of a number of formats, then parses the data forre-formatting into an xml-type synchronization index. This xml-typesynchronization index is then communicated to the mobile computingdevice, and the logic of the mobile computing device software operatesto display the text of the synchronization index on the viewing screen.

The mobile computing device software includes logic to present the textof the synchronization index to the user in a display of a multiplicityof text lines. When viewed in synchronicity with multimedia content, thetext display is rendered in a smoothly scrolling teleprompter-typedisplay. To denote the point within the multiplicity of text lines thatcorresponds to the multimedia, there is a position indicator. Anappropriate placement of the position indicator is approximately fivelines of text from the top of the text display. The position indicatormay be a rectangular, semi-transparent graphical element of apredetermined color that is layered beneath the text. The purpose of theposition indicator is to allow the reader to easily and quicklydetermine, when multimedia is playing, the line in the transcript thatcorresponds at a given point in time.

As one display option, the text scrolls, while the position indicatorremains static. As the end of a line of text is reached within themultimedia, the multiplicity of text lines smoothly scrolls tore-position the next line of text into the area of the positionindicator, while the position indicator remains static.

Alternatively, the display of a multiplicity of text lines may remainstatic while the position indicator descends through the text one lineat a time. In this view, the position indicator begins at the top of theblock of text lines and descends through the lines. When the positionindicator reaches the bottom of the block of text lines, the block oftext lines is refreshed, and the position indicator begins again at thetop of the refreshed block of text lines.

With the scrolling text view, the text is synchronously displayed withthe multimedia. The user may perform a gesture to scroll through thetranscript. For example, on an iPad, a bottom-to-top drag (or swipe orflick) on the transcript will scroll to succeeding text lines or pages.This is similar to the way users may be accustomed to scrolling throughtext on a web page delivered within the Safari web browser present oniPad and iPhone mobile computing devices, either using the touchsensitive viewing screen or a multi-touch surface apparatus to acceptgestures.

The user may select settings to determine the behavior of the video whenbottom-to-top swipe is performed to read succeeding transcript pages. Inthe first setting, if multimedia is playing and a bottom-to-top swipegesture is performed, the multimedia will pause, and the transcript'sscroll to succeeding pages will be controlled by succeeding verticallyoriented swipes or gestures. Because the user has now navigated awayfrom the point at which the video was paused, the user may elect tobegin playing the multimedia that corresponds to the text now residingwithin the position indicator, or, in the alternative, the user canelect to “re-sync” the transcript to where the multimedia was paused.This election is made in one or more graphical buttons, which may beplaced on or near the rightmost portion of the position indicator. Forexample, when a user has navigated downward in the text, thereby pausingthe video, one button may be labeled “play here” while another buttonmay be labeled “re-sync/go back.” The “play here” button causes the textto become most important, meaning that the text within the positionindicator will cause the video to scrub to the correct location andbegin playing. The “re-sync/go back” button will cause the video tobecome the navigational aid or bookmark, causing the video to resumeplay where it was paused, and the transcript will be refreshed to placethe correct text back into the position indicator. With these twooptions, a user can use the text to navigate the video, or the video tonavigate the text.

The user may also elect to begin multimedia playback from any line oftext, even if that line is not congruous with the position indicator. Todo so, a preselected gesture applied to a line of text is assigned toplay multimedia. This gesture may be a swipe, a tap, a double-tap, aswipe and tap, a triple-tap, or any other gesture the user may select toassociate with the playback function. This is helpful, for instance,when a user is playing multimedia and desires to view other text, butkeep the multimedia playing in the meantime. The user may set a settingto, upon a gesture to view succeeding pages of text, keep the multimediaplaying. While that media is playing, albeit now not in synchronous viewwith the transcript, a new play location can be found with the text, andthe new multimedia playing location can be accessed with the selectedgesture. While the video plays, the user can use the “re-sync/go back”button to instruct the text to re-coordinate with the playing positionof the video.

Next, the mobile computing device software includes logic for facilityto scrub through the text and multimedia via a navigational scrub bar.This navigational scrub bar is preferably visible at the bottom of thetext display. It comprises a graphical representation of a time durationt=x to t=z, and a graphical element as a place marker to mark theposition t=y between t=x and t=z. If the navigational scrub barcomprises the entirety of the video, x=0 and z=end. If the navigationalscrub bar does not comprise the entirety of the video, but only aportion thereof as adjusted by the user to focus on less than all of theavailable video, t=x is earlier than t=z. The logic enables the user toscrub through the video and text by adjusting the location of the placemarker. The place marker can be moved by corresponding input on a touchsensitive display screen, for example by using a finger to slide theplace marker to a location on the navigational scrub bar.

Advantageously, when the place marker is moved, the logic achieves nearsimultaneous coordination between the multimedia being displayed and thecorresponding text. To the user, the two appear to be inextricablylinked, providing a seamless integration between scrubbing text andmultimedia on a mobile computing device. Such near simultaneouscoordination between the multimedia being displayed and thecorresponding text is preferable in the circumstance where themultimedia content resides in memory on the mobile computing device,which the mobile computing device's processor can access with lowlatency, as opposed to the instance when multimedia is being streamed tothe mobile computing device. For streamed multimedia, inferior datacommunication speeds may temporarily interrupt a seamless video display,in which case temporary reversion to an audio-only stream may stillprovide the near simultaneous coordination between media and text theuser desires.

The term “near simultaneous coordination” should be understood to be assimultaneous as can be achieved given the computing constraints of aparticular mobile computing device and, described above, the speed atwhich data can be transmitted to the mobile computing device. In someexamples, the multimedia resides on the mobile computing device and thecoordination between text and multimedia may be a fraction of a second,a delay length being virtually imperceptible to the user. The lower thelatency able to be achieved, the better the user experience.

The function of the logic for the navigational scrub bar may provide aheretofore unavailable user convenience in simultaneously navigatingsynchronous text and multimedia on a mobile computing device.

Next, the mobile computing device software includes logic for facilityto select a word or range of words, apply certain attributes to them,and save those attributes. For example, in a deposition transcript, asdescribed above, the user may select a “page:line to page:line” range,assign a color to it, assign it an issue name, and attach a note to beassociated with the selected text. This process is known in the legalindustry as issue coding a transcript.

In some implementations, on the Apple iPad a double-tap gesture on aword will cause a pop-up with handles to refine the range of the text tobe selected and with options for the user. Those options include abilityto select a color, assign an issue name, create a new issue name,provide a note, copy, copy to an email, and so on. Generally, the useris provided facility, via a series of pop-ups, to perform a multiplicityof annotations or edits to the range of text selected.

These transcript manipulations can be saved to the memory of the mobilecomputing device, and they can also be communicated back to the webapplication on the hosting server. Later, succeeding users may accessthe revised synchronization index, thereby achieving a collaborativework environment.

Once an annotation has been made, for example a range of text beinghighlighted in the color yellow and assigned to the issue group“Plaintiff's direct designation,” that yellow highlight will persistwith that text any time the user has selected its display. In addition,since that text corresponds to a discrete portion of multimedia, theuser may optionally be provided the facility to locate that yellowannotation by reference to the navigational scrub bar as well. Withreference to FIG. 19, another graphical element, for example a hatchedrectangle, labeled 30, may be provided either on the navigational scrubbar, or above it, or below it to indicate a corresponding annotation inthe text. In this example, the user may look to the navigational scrubbar and would see a hatched rectangle that corresponds to thehighlighted text. In this fashion, the user can quickly determine bylooking at the navigational scrub bar to determine where highlightedportions of the text reside, without necessarily needing to view orscroll through the entirety of the text transcript. As shown in FIG. 19,other graphical elements (e.g., graphical element 31) may be displayedto indicate other highlighted/designated portions of the texttranscript. The graph elements 30 and 31 may be displayed differently toindicate that the highlights/designations were made by different users,that the highlights/designations correspond to different issues, or thatthe highlights/designations were made at different times.

Further, the mobile computing device software includes logic forfacility to sequence pre-selected “page:line to page:line” ranges oftext and associated multimedia, to the exclusion of other text. Forexample, a user may have ten different issue codes (with each beingassigned its own color), each having a multiplicity of page:line rangesassociated with them. The user may elect to view only those page:lineranges that correspond to, say, issues 1, 3 and 7. In other words, anytext not included within those issues would not be seen. Thecorresponding multimedia plays as well, skipping those portions of textnot selected, and jumping as seamlessly as possible to the multimedialocation corresponding to the succeeding selected text. As aconsequence, all visible text would have an associated highlightingcolor, and the entirety of the refreshed navigational scrub bar wouldhave at any discrete time at least one color. In this fashion, the usercan very quickly determine from the navigational scrub bar the total runduration for the selected issues, the relative time duration expended byeach discrete issue, concentrations of one issue relative to anotheralong the time spectrum, and so on.

Sequencing of video clips also allows the attorney to very closelyapproximate the multimedia he intends to display to the judge or a jury,with the advantages being that no cumbersome trial presentation softwareis required, that no video editing expertise is required, that only amobile computing device is required, that the multimedia can be sharedwith other users, say a legal client, because the multimedia is capableof being streamed to any validated mobile computing device by thestreaming server, and so on.

In some examples, once the user has selected that sequence of videoclips desired, the logic of the mobile computing device may beconfigured to output a load file intended to be used for creating videoclips within trial presentation support software applications, such asSanctionII with a StartPage.StartLine+EndPage.EndLine format.

In some implementations, and as described above for the web application,the mobile computing device software also includes logic to calculate aduration list, without necessity of creating video clips, by performingcalculations from the synchronization index itself. For example, if auser annotates a page:line range within a transcript, the softwareexamines the synchronization index to compute the time differencebetween EndPage.EndLine and StartPage.StartLine. In this manner,computations can be output for each annotation, for all annotations thatcomprise one issue, for merged issues, for more than one issue, and forthe totality of all annotations. This may be particularly useful whenused in conjunction with the above described logic for facility tosequence pre-selected ranges of text and associated multimedia.

As described above for the web application, the mobile computing devicesoftware also includes logic to perform text searches. The user mayelect to perform a text search on one synchronization index, on amultiplicity of synchronization indexes, or on an entire universe ofsynchronization indexes.

The mobile computing device software may also include logic to determinetext font, size, color, and other attributes. In the context of a legaltranscript, it may be particularly important that any such changes notaffect the user's ability to determine the correct page:line referenceshould that portion of the transcript need to be cited in a legal paperor to the court. One way to ensure line breaks of the originaltranscript are ascertainable, even if font sizing causes a line wrap, isto insert a selected character, such as a relatively unobtrusivered-colored pipe character, at the end of each line which, when touched,reveals a page:line reference. In this manner, font size can beincreased to cause line wraps but the user can still simply access apage:line reference.

Moreover, it may be advantageous to employ logic to ensure thetranscript can be read in a comprehensible format. For example, in alegal transcript, hard returns between lines of text attributable to acertain speaker should be retained. A line tagged as a question by anattorney, often begun with the characters “Q:” should never be confusedwith the witness's answer, which lines may begin with the characters“A:” or with an indication of the witness's name. To the extentpossible, the logic should parse questions from answers, and perhapsallow the user to apply certain attributes to only certain portions ofthe transcript. An example would be a deposition transcript, where thereader may more easily read and understand the content if all questionsor attorney statements are in bold text, whereas all statements oranswers by the witness remain in a non-bold type font. Other defaultfont characteristics may be desirable and may be specified within thesynchronization index, for example tagging all speech by one user as apre-selected color.

The mobile computing device software may also include logic to receiveonly that content that a user is authorized to receive. A user of amobile computing device may utilize a synchronization index to view thetext of a deposition transcript. In some circumstances, attorneys maydesignate certain portions of the transcript as Confidential, RestrictedConfidential, or Attorney's Eyes Only, categories typically defined bythe court in a protective order that limits the individuals who mayrightfully access certain testimony. In these instances anadministrator, via the web application, manages a set of permissionsunique to each user within the administrator's workgroup. In thisfashion, an administrator may utilize the synchronization index to issuecode those transcript sections that might be, for example, Confidential,Restricted Confidential, or Attorney's Eyes Only. A subsequent user maynot be assigned full administrative privileges, but may be grantedaccess to the transcript including sections issue coded Confidential,but excluding sections coded Restricted Confidential or Attorney's EyesOnly. For this user, the logic of the mobile computing device softwaremasks or otherwise redacts those portions of the text that correspond totranscript sections he is not authorized to see. In coordination, thelogic also prevents that user from accessing multimedia that wouldcorrespond to transcript sections he is not authorized to see.

In circumstances where the video upon playback reaches a point for whichaccess is limited by, say a Restricted Confidential designation, thevideo would seamlessly play while omitting the area of limited access.In other words, video playback stops at a section for which access islimited and immediately resumes at a point after that section of limitedaccess. Optionally, an audio indicator of limited duration, for examplea short ping, can be inserted to audibly indicate the omission ofcertain multimedia content and corresponding text.

Relatedly, the mobile computing device software may also include logicto ensure the delivery of multimedia and synchronization indexes only toauthenticated users by an authentication routine, which may includeencryption. That is, the mobile computing device, before receiving asynchronization index or multimedia from either the hosting server orthe multimedia server, needs to be authenticated in an authenticationprotocol. Thereafter, the content may optionally be encrypted andcommunicated to the mobile computing device. Such security may beimportant in certain circumstances, such as legal proceedings where theparties need to ensure access is provided only to specific individuals,and that any access can be logged and verified as having occurred. Dataencryption techniques may be employed to achieve high levels of securityand accountability.

Next, the mobile computing device software may also contain logic toplay multimedia at a speed that is greater than normal speed. It ispossible for multimedia to be consumed and understood more easily, andwith a higher level of retention of the material, if a user reads textat the same time audio is available. The logic that controls playbackspeed allows the user, via a GUI, to select a playback speedincrementally faster, or slower, than a regular playback speed (e.g.−2.0×, −1.9×, −1.8×, −1.7×, −1.6×, −1.5×, −1.4×, −1.3×, −1.2×, −1.1×,1×, 1.1×, 1.2×, 1.3×, 1.4×, 1.5×, 1.6×, 1.7×, 1.8×, 1.9×, 2×, where x isthe normal playback speed.) The user may also be provided the option tokeep the pitch of the video equivalent to that of normal playback speed,which aids the user in comprehending human speech within a regular pitchrange, but at increased speed. To accomplish faster than normal playbackspeeds, the multimedia may be played from memory resident on the mobilecomputing device, for example in flash memory or cached in RAM frommultimedia that is streamed to the mobile computing device from amultimedia streaming server. With availability of sufficient datatransfer speeds between a multimedia server and the mobile computingdevice, faster than normal playback speeds may also be accomplished.

The logic of the mobile computing device software may also be configuredto negotiate among several multimedia variant playlists to select themost appropriate bitrate and/or video dimensions depending on the mobilecomputing device of the user, the wireless connectivity speed available,and the playback option selected. For example, if the user's mobilecomputing device has limited connectivity speed and the user has electedto play the multimedia at 2× speed, the multimedia server may not beable to deliver the multimedia at high bitrate and large video dimensionwithout pauses, dropouts or the like, while the multimedia caches on themobile computing device. Accordingly, logic of the mobile computingdevice software may negotiate down to a low-bitrate audio only streamavailable within the variant playlist, which facilitates the ability ofmore content to be cached to the mobile computing device more quicklythereby providing the user uninterrupted playback of audio at hisselected 2× playback speed.

Next, the mobile computing device software may also include logic tocause the multimedia to play in full screen mode upon a gesture and,upon subsequent gesture, retreat back into a pre-configured location foruse with synchronous text display. If, for example, the viewing screenis apportioned according to the layout of FIG. 3, the multimedia displayarea resides in a fixed location at the upper left of the viewing area.To send the video to occupy the full viewing area, a gesture, forexample a pinch out (a two finger reverse pinch), will expand the videoto full screen. To return the video to its original location, a pinchgesture, or similar, may be used.

The mobile computing device software may also include logic configuredto receive a realtime text feed (or other temporary or draft transcriptwithout having been proofread and corrected for errors) from a courtreporter or stenographer during the taking of testimony, for exampleduring a deposition or a court proceeding. Common computer assistedtranslation (CAT) systems may be configured to broadcast a realtime textfeed, including via a wireless LAN, and the logic of the mobilecomputing device software may be configured to wirelessly receive suchbroadcast. The realtime feed is then displayed on the viewing screen ofthe mobile computing device, and the annotation functionality, asdescribed above for issue coding, may be applied to the drafttranscript. It may be advantageous to utilize this logic within anoverall system for transcript management, where the temporarytranscript, and any user annotations on a mobile computing device, arecommunicated back to a web application. Later, the temporary transcriptcan be replaced by the final transcript in the manner that theannotations made on the draft will match the correct location within thefinal transcript. Thereafter, the final transcript can be delivered backto the user for use as a synchronization index, as has been describedthroughout.

The logic configured to receive a realtime text feed may allow asubstantially wire-free deposition workspace, it allows a court reporterto conveniently carry and distribute mobile computing devices todeposition attendees to receive a realtime text feed, it allows an easeof use and ease of configurability not presently available in the courtreporting industry, for example by use of an intuitive annotationfunctionality user interface, it makes consumption of realtime texttranscription services more simple, thereby increasing the likelihoodthat such services will be utilized (and thereby providing additionalrevenue for court reporters with the professional ability to transcribeat realtime speeds), and it meets the further long felt but unmet needof providing an overall enhanced user experience for receipt of realtimetext transcription services on a mobile computing device.

Each of the aforementioned elements of logic for operation on a mobilecomputing device may enhance the overall user experience, with thecombination of such logic providing heretofore unavailable functionalityfor use of text-synchronized multimedia on a mobile computing device.The user's workflow and efficiency in reviewing transcripts on a mobilecomputing device may be greatly improved.

While examples have been often described in the context of textsynchronized multimedia delivery for legal testimony and correspondingvideo, the described techniques may be adapted for many other commercialuses and is not intended to be limited for use within the legalindustry. By way of non-limiting examples, the following applications ofthe described technique are intended to be possible without departingfrom the spirit of the disclosure.

Other Applications of the Embodiments

Music

The sale or rental of music on mobile computing devices accounts forbillions of dollars of revenue for those who have developed interfacesfor distributing music and other multimedia. For example, Apple Inc.'siTunes software has found considerable commercial success because itprovides a positive user experience and provides convenient access to anextensive library of multimedia content for purchase or rental. In lightof that commercial success, Apple and its competitors need features toset their multimedia distribution systems apart from others, making themmore interesting or more functional for users.

For example, the overall user experience sways a consumer and hispurchasing habits. It is frequently the case that a convenient systemwill find more users than an inconvenient one, and a system with manyfunctional options for the user will find more users than one withlimited user options. Each additional feature has an impact on overallcommercial success. Apple's CoverFlow within iTunes (conceptuallyexpanded upon within the Apple OS-X Snow Leopard operating system), forexample, simply makes album art, the graphic image comprising the frontof a record album or the cover of a CD's packaging, available to theconsumer as a method to quickly shuffle from one album, or set of songs,to another album. This functionality is said to enhance the user'soverall experience with using Apple's mobile computing devices—iPods,iPhones, iPads and the like—and thus has increased the likelihood that aconsumer will purchase multimedia for use on those devices as well asthe devices themselves.

Accordingly, an object of this aspect of the present disclosure is toenhance the overall user experience in using or consuming multimedia byproviding a synchronization index to accompany multimedia, providing theuser an ability to smoothly and substantially synchronously scrubthrough the multimedia using written words as a roadmap. It may beadvantageous, for example, to offer a system for a mobile computingdevice that allows lyrics to be delivered with music in a fashion that aparticular line in the lyrics can be used to advance the music to thatspecific point within the song. The described techniques contributegreatly to the functionality of a multimedia delivery system, such asiTunes or its competitors, and therefore has a very positive impact onthe overall user experience, ultimately translating into increasedcommercial success by way of sales of multimedia and the mobilecomputing devices sold to play the multimedia.

By way of example for the music industry, an artist records a song andthe song is encoded into a common electronic format, including .mp3,.wav, AAC or a number of others. A software application can be used as asynchronization index generator to synchronize the lyrics to the songand output a synchronization index. The synchronization index, ordatabase structure, may be a formatted .mdb file, with each lyric linereceiving a timestamp corresponding to the time at which a particularword is heard. Using the web application, the mdb format is parsed andreformatted into an xml-type format. The synchronization index andinformation about the song to which it is associated is communicated toa remote server, for example a server accessible by Apple's iTunes, orthe web application of the present disclosure. A user then uses a mobilecomputing device to communicate with the remote server to access thesynchronization index and retrieve the associated song.

Once the synchronization index has been communicated to the mobilecomputing device, a hierarchical seek order may be employed by themobile computing device to retrieve multimedia. For example, when asynchronization index refers to a song, that song may reside on themobile computing device already. The software on the mobile computingdevice first inspects the mobile computing device for the presence ofthe song. If the song is found, it is played with the synchronizationindex. If the song is not found resident on the mobile computing device,a subsequent step may be to seek for the song resident on a devicewithin a network accessible by a WiFi connection (for example a localarea network of computers within one's home), and if found, communicatedfrom that device on the network to the mobile computing device forplayback. Next, if the song still has not been found, the software mayseek for the song in a remote location accessible by internet connectionor accessible by 3G or telephone/data network, and so on until thecorrect song can be located and delivered for playback on the mobilecomputing device. A user may set preferences to choose the highestbandwidth connection for highest quality multimedia.

As the song plays on the mobile computing device, lyrics display in asynchronous, smoothly scrolling teleprompter-type fashion. Each line oflyrics is, preferably, linked to a specific time within the song. Apredetermined gesture (e.g., a swipe, a double-tap, a triple-tap, orsimilar, or movement of the line of lyrics to a static positionindicator to indicate a desired playback location) jumps the song tothat time of elapsed duration.

Some implementations include a multiplicity of synchronization indexesthat are aggregated for search to provide an extensive library ordatabase of text-searchable song lyrics. One example is an album ofsongs and corresponding lyrics, as one may find printed upon a foldedinsert accompanying music distributed on a CD. Another example is alibrary of lyrics, comprising hundreds or thousands of songs, adaptedfor use and search from, say, Apple's iTunes. For example, a user maysearch for a particular phrase from a song. The library of lyrics wouldbe searched for a match, and the matches, with the synchronization indexor a portion thereof, would be returned to the user. Optionally, thesearch result may also include data such as an artist name, a songtitle, an album name or image, a recording date, and so on to allow theuser to locate a particular song. Because a synchronization index isreturned, the user may jump directly to the specific part of the songassociated with the lyric in the synchronization index, and a songpreview may begin to play from that point for a pre-determined, shortduration that is less than the duration of the entire song.Advantageously, the user may more easily determine if that is the songdesired, and then complete a purchase, a download or a rental of thesong. The user may also be exposed to multiple artists' renditions ofthe same song, allowing more song purchase options for the user.

In some examples, a search for text would access a complete text library(e.g., hosted by Apple, Google, Microsoft, Amazon, or similardistributor of large libraries of multimedia content), return the hits,determine whether the identified song resides locally on the device,and, if not, provide a short stream of the song, with a solicitation topurchase the song or album. The same functionality applies toaudio/visual books and movies, among other uses, without departing fromthe scope of the present disclosure.

Moreover, use of the described techniques for music may overcome thepitfall of karaoke-type closed captioning that occurs as a result of alimited number of text words being able to be displayed at once, whereasin certain circumstances a more complete text transcript would bepreferable. For instance, song lyrics have the characteristics ofpoetry, or a complete story. The context of the words within the overallcomposition may have meaning to the artists, or to the reader, when readas a whole, rather than one line at a time or a few words at a time.

In addition, copyright holders for music may find it advantageous todistribute music with a synchronization index as opposed to just themusic alone. Users are provided enhanced functionality with the abilityto interact with and comprehend lyrics. Copyright holders are presentedwith a new opportunity for a revenue stream, wherein delivery of asynchronization index with music may command a monetary premium over themusic alone, as well as a way to discourage uses of non-authorized worksor content. If a user is certain the synchronization index purchasedfrom the copyright holder of the music is guaranteed to be accurate intextual content and in the time links utilized to achievesynchronization between text and multimedia, the user may avoidalternate works that do not have such guarantees.

The described techniques may solve the problem of conveniently locatingsong lyrics from a mobile computing device, and then achievingsynchronous playback of a song and its lyrics on a mobile computingdevice.

Movies and Motion Pictures

In another implementation, the multimedia may be a film, movie, motionpicture, theatrical release, musical, music video, movie preview ortrailer, or other video generally consumed for entertainment or used fordocumentary purposes. The synchronization index in this examplecomprises a screenplay, which may have spoken text, song lyrics, andother non-spoken words that the author may intend to be displayed, forexample a screenwriter wishing to provide the audience the experience ofreading his written direction cues for display with the video for studyof how such cues were theatrically executed in the film itself.Screenplay authors can provide heretofore unavailable convenience forfilm industry professionals as well as new audiences. Screenplay authorsgenerally do not have a broad consumer base for their written workproduct outside of industry-specific professionals, and the describedtechniques may allow a new consumer base to appreciate those writtenworks of authorship, all with the additional benefit of beingsynchronously viewed with the resultant movie and theatric execution.

Moreover, another feature is provided by the present disclosure in thatDVDs, Blu-ray discs and the like have typically been navigated bychaptering information included on the recording media that can be readby playback devices. These chapters have historically been used tonavigate to preselected portions within the movie, such as maycorrespond to an Act or a Scene. For example, a movie may have containedten or twenty discrete chapters, but typically not many more, andcertainly not an order of magnitude more, which possibility is providedhere for the first time.

With streaming media becoming more prevalent, however, consumers requiresomething other than mere chaptering information for convenient accessto the multimedia. With the described techniques, the synchronizationindex presents opportunity to apply timestamps to every word, creatinghundreds, or thousands of “micro-chapters” that the user can use fornearly immediate, non-linear access to virtually any point within amovie. Legacy chaptering information can also be retained, as thesynchronization index is not limited to spoken words, giving theproducer a complete palette of navigation options to provide theconsumer, including a table of contents and full-text search-ability.

Another advantage of the present disclosure is that the system allowsthe ability to text-search entire multimedia libraries. This allows auser to quickly access and watch selected scenes, with the result beingthat users rely more upon a cloud computing-type content distributionmodel wherein the entities providing those services, often on asubscription based business model, will retain paying members and theresultant revenue stream. The system also enhances the user experienceof consuming multimedia, particularly on mobile computing devices, afactor that tends to increase market share for multimedia contentservice providers, such as Netflix or Apple sales through iTunes.

In addition, the system allows search criteria to be applied acrossmultiple films with content from the synchronization index having thepotential to be associated with other metadata for search purposes. Forexample, it is contemplated that a user may perform a search based on anactor's name, or a character's name, and a line spoken within a movie,without the user having to necessarily recall or search for the name ofthe movie itself. A search result, with appropriate metadata and taggingapplied to the synchronization index, allows a user to access the exactscene desired.

In contrast, a common model for such searches is a search performed onthe YouTube.com website for hosting video. With the YouTube model,millions of video clips are hosted which have been posted by thousandsand thousands of disparate users, and the searcher finds the relevantscene only if the user posting the content happened to add the correctwords to the title of the clip or as keywords associated with theuploaded video. The YouTube model for finding relevant and interestingvideo content, particularly for commercial films, is rife with copyrightinfringement and the potential therefor, with content owners needing tovigilantly police a multiplicity of hosting locations containing theirown material and then follow procedures with those website operators toseek removal of the material due to copyright violations. Movie studiosand other copyright owners are faced with the unpleasant prospect ofhaving the content removed, thus potentially alienating a portion oftheir fan base who view content within those online communities andsocially interact with other users, or leaving the content available butsuffering economic loss due to 1) advertising losses, in thatadvertisers pay hosting services like YouTube based on the trafficgenerated, whereas that traffic often would not exist but for multimediacontent users' desire to view interesting content, such as newlyreleased films and so on; 2) the inability to control the user'sexperience; 3) the inability to display advertising of the copyrightowner's choosing, be it for other movies it produces or distributes, forgoods, for services, or for any other type of promotional, branding orgoodwill enhancement; 4) lost licensing fees; 5) social networkinginteractions built upon common interest in movies, music, or othermultimedia; and so on.

The described techniques provide a multimedia copyright owner thefacility to make an entire film available as streaming media to a mobilecomputing device, and searchable via the synchronization index. Once auser performs a search, the user's identity can be verified to determineif he ought to be granted full access to the multimedia, say by having acurrent paid subscription with the multimedia content provider. If theuser has no subscription, and thus ought not be provided unlimitedaccess to the multimedia, it may be advantageous to offer the userlimited access, or preview access, to, say, a 30-240 second clip only,thereby reducing incentive for searchers to hunt “unauthorized” contenton unauthorized distribution channels. The described techniquesincentivize users to maintain paid subscriptions to content providers inreturn for convenient, searchable, “always on” full access to theircontent. The modern mobile computing device user desires such convenientaccess.

Moreover, for the system of the present disclosure it may be desirableto increase the amount of multimedia with synchronization indexes. Onemay leverage existing synchronization index-type information for usewith the system of the present disclosure by de-multiplexing closedcaptioned data from multimedia, or by transcoding existing closedcaptioned data sets. In this fashion, the web application, by the fileformat and data parsing logic 6-8, may re-purpose closed captioned dataformats into an xml-type synchronization index. Closed caption filetypes suitable for this transformation include: SAMI (SynchronizedAccessible Media Interchange), SMIL—(Synchronized Multimedia IntegrationLanguage), SubViewer (*.SUB), SubRip (*.SRT), .SBV, Adobe Encore DVD(*.txt), Advanced SubStation Alpha (*.ass), AQTitle (*.aqt), Captions 32(*.txt), Captions DAT (*.dat), Captions DAT Text (*.dat), Captions Inc.(*.txt), Cheetah (*.asc), CPC-600 (*.txt), DKS Subtitle Format (*.dks),DVD Junior (*.txt), DVD Subtitle System (*.txt), DVDSubtitle (*.sub),FAB Subtitler (*.txt), (Author Script (*.txt), Inscriber CG (*.txt),JACOSub 2.7+ (*.jss; *.js), Karaoke Lyrics LRC (*.lrc), Karaoke LyricsVKT (*.vkt), KoalaPlayer (*.txt) (equal to one of the variations ofTMPlayer), MAC DVD Studio Pro (*.txt), MacSUB (*.scr), MicroDVD (*.sub),MPlayer (*.mpl), MPlayer2, (*.mpl), MPSub (*.sub), OVR Script (*.ovr),Panimator (*.pan), Philips SVCD Designer (*.sub), Phoenix JapanimationSociety (*.pjs), Pinnacle Impression (*.txt), PowerDivX (*.psb),PowerPixel (*.txt), QuickTime Text (*.txt), RealTime (*.rt), SAMICaptioning (*.smi), Sasami Script (*.s2k), SBT (*.sbt), Sofni (*.sub),Softitler RTF (*.rtf), SonicDVD Creator (*.sub), Sonic Scenarist(*.sst), Spruce DVDMaestro (*.son), Spruce Subtitle File (*.stl), StreamSubText Player (*.sst), Stream SubText Script (*.ssts), SubCreator 1.x(*.txt), SubRip (*.srt), SubSonic (*.sub), SubStation Alpha (*.ssa),SubViewer 1.0 (*.sub), SubViewer 2.0 (*.sub), TMPlayer (*.txt) (fivedifferent variations), Turbo Titler (*.txt), Ulead DVD Workshop 2.0(*.txt), ViPlay Subtitle File (*.vsf), ZeroG (*.zeg).

Because existing closed captioned data sets may be old or inaccurate,users may modify the transcriptions as one would in making erratacorrections to a deposition transcript. It may be advantageous to employan open, wiki-style ability for a multiplicity of users to correcttranscriptions. The result is that users are drawn in ever closer to themultimedia that attracts their interest, benefitting multimedia ownersand creators as well as subsequent users and consumers.

As is evident, the present described techniques may contribute to anenhanced overall user experience, and may enhance commercial success andviability with respect to multimedia sales, content deliverysubscriptions, and mobile computing device sales.

Educational Content and Regulatory Compliance

Yet another use for the described techniques is use for educationalmultimedia content and for access to multimedia created by ordistributed by government, including multimedia whose distribution mustmeet rules and regulations for accessibility by disabled persons. Therepresently exists a large body of multimedia that is configured to meetrules and regulations for accessibility by disabled persons, howeverextensive use of closed captioning, with its many shortcomings set outabove, for such compliance does not adequately address viewership ofsuch multimedia on mobile computing devices. Mobile computing devicesoften have viewing screens that may be too small to accommodate legibletext superimposed over video, they are often simply not configured toreceive such closed captioned text, and the video files themselves whenbeing transcoded or re-formatted for streaming media deliveryunnecessarily strip or remove closed captioned text that may haveoriginally provided regulatory compliance.

With the described techniques, multimedia content providers can providea heretofore unavailable convenience for disabled users of mobilecomputing devices in a way that meets the requirements and intent ofSection 508 Amendment to the Rehabilitation Act of 1973 (29 U.S.C.§794d), and others.

Educational content distributed by schools, universities, researchinstitutions and the like can be consumed in multiple languages, as asynchronization index may allow the user to elect one of severallanguage settings. Additionally, comprehension of content can be vastlyimproved as many studies indicate comprehension levels increase whenusers hear and read content at the same time. Moreover, the describedtechniques may facilitate learning, literacy initiatives, educationalinstruction, and they do so in a manner that mobile computing devices,with their increasing ubiquity, can be utilized for such worthwhilesocietal betterment objectives.

Voice-to-Text Voicemail and Telephony Recording

In another example, the multimedia may be a voicemail and thesynchronization index may contain a transcription thereof. With theincreasing use of mobile telephones/smartphones and voicemail services,many users find themselves deluged with lengthy voicemail messages thatthey do not want to take the time to listen through. New businesses haverecently attempted to help solve the problem by applying voice-to-texttranscription of voicemail and then sending, for example by email, thetranscription of the voicemail to enable the recipient to more quicklyunderstand what the speaker intended to convey. This transcription isoften used in conjunction with a “visual voicemail” system.

With the described techniques, users are no longer required to choose tolisten to the message or to read the transcription as an either/orproposition, but rather they now have for the first time a convenientway to navigate the voicemail multimedia with a synchronization index ona mobile computing device. For example, voice-to-text transcriptionsoftware may be employed to create a transcription and in so doing itmay apply a timestamp for each word, and in some instances one timestampfor what the software perceives to be the beginning of a word and onetimestamp for what it perceives to be the end of the word. Becausevoice-to-text transcription software engines do not render perfecttranscriptions, for a host of reasons, it may be important for therecipient to access the voicemail, but not necessarily from thebeginning or with cumbersome fast-forward and rewind transport controls,to confirm the content of the message the sender intended. This is notunlike the use of an errata sheet in the legal context, wherein a userneeds to check the transcription for accuracy; a task that is mucheasier to accomplish with convenient, non-linear access to theunderlying multimedia itself, and preferably on a mobile computingdevice. Use of the described techniques for the aforementioned purposemay overcome many shortcomings of existing voicemail and transcriptionsystems and services. It is another example of an enhanced userexperience and provision of a solution to a long felt but unmet need forthe mobile computing device user.

Use of the Synchronization Index and a Mobile Computing Device toControl Multimedia Delivery to a Receiving Device

In some implementations, the synchronization index communicated to themobile computing device may be used to seek video from a remote server,where the remote server is then asked to begin delivering video at timet1 to a device other than the mobile computing device itself. Forexample, a consumer of cable or satellite television services may desireto scrub video content that he can watch on a television within hishome. In this embodiment, the user views the transcript, or asynchronization index, on the mobile computing device (e.g. Apple iPad),gestures on a transcript display on the iPad intending to scrub video toa certain location, then the request is communicated to the server toinstruct it to deliver video at time t1 to destination d1, for example,a selected television within the home that has multimedia contentcommunicated to it via a set-top DVR-type device, then the multimedia isdelivered and, optionally, cached in whole or in part. In this manner,the iPad with the synchronization index functions as a sophisticatedtype of remote control for video content, using a transcript of themedia to navigate precisely to a desired viewing location.

In common digital video recorder (DVR) devices, a user can elect torecord broadcast television programming to a hard drive integral to theDVR. A user may navigate to a program guide and select content to recordat a future time, or the user may elect to record the content beingviewed live. Users then play back the multimedia at a later time. Duringplayback, most DVR devices facilitate the ability to fast-forwardthrough the recorded content, for example to avoid commercials, and theability to rewind. Both fast-forward operations and rewind operationsoften include variable rate fast-forward and variable rate rewind,allowing the user to fast forward at differing speeds (e.g. 2×, 3×, 4×,−2×, −3×, −4×). These variable rates are intended to allow the user toscrub the video content to access the desired portion of multimediacontent. They remain cumbersome in use, however, because they do notallow pinpoint scrubbing to a specific time location, nor isbookmarking, annotation, commenting, or social networking facilitated inmost DVR-type devices. In further contrast, use of a synchronizationindex facilitates navigation in a multiplicity of ways, includingnavigating to a chapter, scene, or act, navigating by text, skippingdiscrete time blocks (30 seconds, 1 minute, 2 minutes, 5 minutes, 10minutes, etc.), and so on.

In some examples, the synchronization index communicated to the mobilecomputing device may be used 1) to control multimedia resident on themobile computing device itself, 2) to control multimedia resident on themobile computing device itself, but then sent for display to a receivingdevice, 3) to control multimedia not resident on the mobile computingdevice itself, but stored on a receiving device (for example, if thereceiving device was a commercially available home theater DVR-typedevice containing memory where multimedia has already been stored), or4) to control multimedia not resident on the mobile computing deviceitself, and not stored on a receiving device, but communicated to thereceiving device by a serving network or multimedia server (for example,if the receiving device had a network connection but little or noresident memory, or if a DVR-type receiving device had memory but hadnot yet had multimedia communicated to it and stored to memory).

As shown in FIG. 20, the display screen of the mobile computing devicecan be partitioned to allow function buttons to be inserted at thebottom of the display. One such button may be configured to cause themobile computing device to begin an instruction set that causesmultimedia to be sent to or to be displayed upon the receiving device.In this manner, the synchronization index can be used to locate adesired viewing location, the desired viewing location can be viewed onthe mobile computing device in video viewing area, and then theinstruction set can be initiated to cause playback on the receivingdevice. One such example would be to use an Apple iPad, with asynchronization index and resident multimedia, that communicates usingthe Apple AirPlay protocol with an AppleTV appliance that, in turn,delivers multimedia to a home entertainment system.

Any number of hardware devices, or hardware/software combinations, maycomprise the receiving device. By way of non-limiting example, areceiving device may be a digital video recorder, “DVR” (e.g., TiVobranded DVR appliance, Scientific Atlanta Explorer DVR device, DishNetwork DVR, Hughes DirecTV DVR, Motorola DVR, Philips DVR, PolaroidDVR, and so on), a personal video recorder, “PVR”, a hardware appliancefor receiving cable television or satellite television services, acomputer, a television, a smart television, an internet enabled TV (suchas the Sony Internet TV), the AppleTV appliance by Apple, Inc., theGoogleTV appliance by Google, a gaming device with facility forreceiving multimedia data (e.g., the Sony PlayStation3, Nintendo Wiiconsole, Microsoft's Xbox360), streaming media players (e.g., RokuStreaming Player, Seagate FreeAgent Theater+HD Media Player, WesternDigital TV Live Plus HD Media Player), home theater appliances withfacility to receive multimedia communication, and optionally to playBlu-Ray formatted media (e.g., as available from Insignia, LG,Panasonic, Philips, Pioneer, Samsung, Sony, Toshiba, Yamaha, VIZIO), amonitor, or another mobile computing device.

Multimedia may be delivered by any multimedia content distributionmethod, including by computer network, cable television or satellitetelevision service, subscription multimedia delivery service (e.g.,Netflix, Hulu, iTunes, Rhapsody, Spotify, MOG, Napster, Rdio,Grooveshark.com, Pandora.com, alluc.org, talk radio subscription contentby radio personality, and so on), peer-to-peer network, Apple AirPlay,Universal Plug and Play (“UPnP”, generally defined as a set ofnetworking protocols for primarily residential networks withoutenterprise class devices that permits networked devices, such aspersonal computers, printers, Internet gateways, Wi-Fi access points andmobile devices to seamlessly discover each other's presence on thenetwork and establish functional network services for data sharing,communications, and entertainment) and similar methods for deliveringmultimedia content.

While the term “video on demand” has historically been associated onlywith the user being able to select the time at which a video may havestarted to play, the described techniques allow a user to scrub videousing a synchronization index, and vice versa, to a specific time withina multimedia file, thereby achieving a heretofore unavailable enhanceduser experience.

In another implementation, one mobile computing device can be used tocontrol multimedia delivery to a second mobile computing device. Anintended example would be in-vehicle entertainment systems where onemobile computing device would be a master serving other slave mobilecomputing devices. In practice, a user may download multimedia to amobile computing device, such as an Apple iPad. That first user, say aparent entering an automobile as a passenger with children who havemobile computing devices of their own, may broadcast multimedia to theother receiving devices (e.g., for this example, another iPad, theentertainment system within the vehicle, a handheld gaming device, orsimilar mobile computing device) via an appropriate communicationprotocol and use the synchronization index to facilitate navigationthrough the multimedia.

As another example, the system can be used for audio books transmittedto a different receiving device, such as a vehicle's stereo system, oranother type of in-vehicle entertainment system with capacity todistribute sound and/or video images. A user may use a mobile computingdevice, such as the Apple iPad, to read a book on the mobile computingdevice. Because the text of that book also functions as asynchronization index to the audio book, the reader may insert abookmark, discontinue reading, move to a different location wherereading is impracticable (such as in a car), access the bookmark as partof the synchronization index, and begin the playback of the audio bookat the precise location where the reader had stopped reading.

Moreover, the synchronization index comprising the text of the book canreside on the mobile computing device, while the audio portion can bestreamed, or streamed and cached, to the mobile computing device, or tothe receiving device. In this fashion, the iPad user may take asynchronization index into a vehicle, and direct audio content to bedelivered and optionally cached directly to the in-vehicle entertainmentsystem. For example, the in-vehicle entertainment system may be equippedwith capacity to communicate wirelessly with a computer network (forexample by WiFi, 3G, 4G, Bluetooth, or other wireless connection, suchas OnStar). This capacity may be combined with computer memory integralto the in-vehicle entertainment system, for example by a hard drive,flash memory or similar storage. As a result, the in-vehicleentertainment system is in communication with the mobile computingdevice. Once a synchronization index, or a specific portion thereof, isaccessed, the in-vehicle entertainment system can begin automaticallydownloading and caching corresponding multimedia for use in the vehicleitself. Once the user has finished listening to the audio, the receivingdevice can communicate back to the mobile communicating device the pointat which listening ceased. A bookmark is inserted into thesynchronization index at that point, and the next time the user accessesthe synchronization index, he can return to the correct point within thesynchronization index to either begin reading or to begin a newlistening session. In practice, the user utilizes the synchronizationindex on the mobile computing device to establish non-linear access tocorresponding multimedia content in a new, novel way.

With this and other uses, new commercial opportunities are provided forthose who control distribution of text-based authored works and formultimedia owners. As an example, consumers have typically purchasedeither an electronic book for reading on a mobile computing device, or,in the alternative, they have purchased an audiobook. Very few, if any,users have purchased both products, particularly with the full featureset of the described system for use on a mobile computing device. Withthe described techniques and systems, synchronization indexes andcorresponding multimedia can be offered for sale as a functional,inter-related, interactive product with annotation capacity, rather thanjust text alone or just multimedia alone.

Advertising

Moreover, television broadcasters dislike the functionality of the DVRthat allows commercial advertisements to be skipped during time-delayedprogram playback.

Digital video recorders have been changing the way television programsadvertise products. Watching pre-recorded programs allows users tofast-forward through commercials, and some technology allows users toremove commercials entirely. Some features of this type have beencontroversial, with some major television networks and movie studiosclaiming they violate copyright and should be banned.

The described techniques may facilitate the exposure to advertising,including from companies whose advertisements would have accompanied theinitial broadcast. By using a portion of the mobile computing device'sscreen, targeted advertisements can be unobtrusively displayed in a waythat does not obscure the viewing screen of the multimedia, and in a waythat subtly and unobtrusively invites the user's interaction.

As illustrated in FIG. 20, the iPad display screen can be divided intomultiple sections. For example, a section may be allocated formultimedia playback if the user desires to view content on the iPaditself, a section may be allocated for bibliographic data relating tothe multimedia or programming content, a section may be allocated fortranscript display and manipulation, and a section may be allocated foradvertising or commercial content.

Targeted advertising may be delivered to the iPad. Criteria for targetedadvertising may include: the user's profile and demographic data,geographic location, media being watched or accessed, media consumptionhistory or previously viewed content, word search history, multimediastored on the mobile computing device, and so on.

The portion of the iPad screen to display advertising can beperiodically refreshed. Advertisements can be text, images, ormultimedia. If the mobile computing device with synchronization index isbeing used in its capacity as a remote control for content on areceiving device, the multimedia can be received on a receiving deviceand played back without advertising displaying on the viewing monitor,while the advertising content remains on the mobile computing device.Additionally, advertisement on the mobile computing device concurrentwith multimedia playback allows the user to interact with theadvertisement, for example by clicking an ad or URL to be delivered to awebsite to receive more information about a company or a product.

For example, the synchronization index may be delivered as metadata toaccompany the multimedia. The synchronization index may also provideinstructions for the targeted, timed delivery of advertisement to themobile computing device. These product placement-type advertisements canbe very attractive to the user, yet subtle in the way they are deliveredand displayed to the user. For example, a television program containinga car chase a given duration into the program may have timed metadata inthe synchronization index for the delivery of an advertisementpertaining to the make and model of the car being used in the program.Product placement within multimedia can now be accompanied with acorresponding overt advertisement. With such an ad displayingunobtrusively on the mobile computing device, it increases thelikelihood that the user will click through the advertisement to bebrought to the website of the advertiser. In this way, the describedtechniques may make it easier for consumers to access informationrelevant to their interests or the multimedia content they access.

Another type of advertisement, used often for advertising TV shows onthe same channel, places an ad overlay on the bottom of the TV screen,blocking out some of the picture, as was identified, above, as ashortcoming of closed captioning. Media companies refer to these“Banners,” or “Logo Bugs,” as Secondary Events (2E). Similar to themethod by which severe weather warnings have historically beendelivered, sometimes these secondary events take up only 5-10% of thescreen, but in the extreme, can take up as much as 25% of the viewingarea. Some secondary events comprise sound or movement of a graphicacross the viewing area, which may be undesirable for the user. Despitethe increasing usage of Secondary Events, advertisers are seeking newmethods, like advertising as described herein, to market their productsin concert with television broadcasts and similar programming withoutsubstantially interfering with the users' viewing experience.Advertisers are losing the ability to invade the home, and consumer'sminds, and are consequently increasingly forced to wait for aninvitation from the consumer. Advertisers and content providers arelearning that advertising content customers will be willing to seek outand receive is more likely to impact purchasing decisions. It is thattype of advertising facilitated by the described techniques.

Social Networking

As yet another potential advantage, the described techniques mayfacilitate social networking around multimedia, and present new ways formobile computing device users to share and recommend multimedia to theirnetwork of friends or colleagues.

One way users are provided the facility to interact is in the way thatannotations can be created to a synchronization index and shared withothers. For example, a user may find the chorus of a particular song tobe emotionally moving, and he may wish to share not just that song, butthat particular part within the song, with his social network. Bycreating an annotation to the synchronization index, and then by sharingthat annotation, a subsequent user may obtain access to the first user'sannotated synchronization index for immediate, non-linear access to thespecific part of the song the first user desired to spotlight fordiscussion.

In practice, a user utilizes the synchronization index on a mobilecomputing device. An annotation is made, and information concerning thepage:line range, issue code, color, attached note, and so on may bestored local to the mobile computing device. This information may alsobe transmitted back to the web application on the hosting server, alongwith identifying information about the user, such as the mobilecomputing device identification number (for example, if an Apple devicethen the identification can be found in a unique “UDID”), time, userlocation, and so on. A subsequent user may access the web application onthe hosting server, examine annotations to which he has been allowedaccess, and elect to download those annotations to his ownsynchronization index. As described above, the user may merge or flattenthese annotations with his own, or he may keep a subset and discard asubset, or a combination thereof.

In this manner, a user can select a group of users whose comments, notesand annotations he wishes to receive, to establish a social network. Forinstance, this functionality may be used by a movie's director,producer, its actors or others to provide notes and commentary, on anongoing basis, to enhance one's interaction relating to the film.Whereas Blu-ray discs became an attractive media for content deliverydue in part to the disc's ability to physically store more datapertaining to the film by way of director's commentaries and so forth,the new feature set described throughout this disclosure may makeinteraction with one's audience an organic, ongoing process. Movie fanscan interact with one another, and filmmakers can interact withconsumers.

Similarly, a user may bookmark a part of a synchronization index, andtransmit that bookmark as a URL-type deep link to a specific time withinmultimedia to a subsequent user. Accessing the deep link delivers asubsequent user not only to the correct multimedia but also to thecorrect location within the multimedia. This is particularly useful in acloud computing environment, where users are increasingly not interestedin managing their own multimedia assets on devices they own so long asthey can access the content in another convenient way, for example bystreamed media over the internet or wireless connection. These streamingmedia services, such as Netflix, provide access to vast libraries ofcontent at an affordable price, and, with the described system, becomean even more attractive way for users to consume multimedia content andshare it, and portions within it, with friends and colleagues.

It is easy to appreciate the multitude of interactions possible betweenusers who enjoy discussing music, movies, television shows, lectures,politics, and all manner of content, and how, with benefit of a mobilecomputing device and synchronization index, those social interactionscan be enhanced.

There has been limited capacity to allow a user of a mobile computingdevice access to multimedia at a precise location within the multimediafor the purpose of engaging in social networking. The describedtechniques may address and overcome many of those limitations.

Other uses for a synchronization index and associated multimediaoperating on a mobile computing device are legion, and may includemovies, music, music videos, news, news archives, audio books,television shows, documentaries, radio re-broadcasts and archives,podcasts, business and corporate events, product launches, corporateleadership speeches, CEO messaging, public disclosures, shareholdermeetings, academic uses, educational uses, travelogues, politics andgovernment, court proceedings, legislative sessions, executive speeches,political campaigns, local governments, school boards, regulatorycompliance, uses for those with disabilities and compliance with theSection 508 Amendment to the Rehabilitation Act of 1973, medical recordstranscription, call centers, training, custom applications, foreignlanguage education and learning, motivational speaking, religiousinstruction, sermons, worship, voice-to-text transcription, voicemail,historical preservation, sports broadcasts, infomercials, commercialadvertising, material of children's interest, and many others.

For example, with commenting features of the present disclosure,crowd-sourced multimedia indexing may be achieved. Large groups of usersmay voluntarily contribute to create a knowledge base for use by others.This same “crowd-sourcing” can be applied to indexing multimedia. Oneuse of crowd-sourcing has been described above in relation to users thattext transcribe multimedia, modify the text of a transcription or toadjust timestamps used to synchronize text to multimedia within asynchronization index.

Another use of multimedia indexing is in advertising, where a product'sproducer seeks to enhance brand awareness. With crowd sourced tagging ofmultimedia, particularly with a synchronized transcript and commentingfunctionality, certain brands may be “tagged” by users.

Product placement in the multimedia content may be supported in theembodiments. A corporation may desire to have its goods used, or evenhighlighted, within a major motion picture and be noticed by the viewer.For example, a producer of goods or services may run an advertisingpromotion in association with a multimedia or movie producer, wherebythe brand owner provides incentives for the consumer to 1) view themultimedia, including for the purpose of finding its products usedwithin the multimedia, 2) “tag” those occurrences of product placement,for example by creating a text comment or other marker (or “tag”, “hit”,“vote”, “find”, “like”, “digg”, “brand-emoticon”, or similar) at thepoint in the multimedia when the product is used or shown, 3) sharethose “tags” or comments with one's social networking contacts, and 4)receive from the brand owner an incentive for having tagged itsproducts.

Contests or promotions may be held for the users who tag and share acertain number of times, or who tag and share a certain number of films.It may be that a product is shown multiple times in a film, and if auser is able to tag a threshold number of them (within a pre-definedtime range of the product being visible on screen), he may be eligiblefor sweepstakes contests, drawings or rewards, such as a coupon for freeor discounted product, or loyalty points to be spent at a certain storefor a certain product. More expensive items may also be used to enticeusers to tag films for product usage. For example, a car manufacturermay offer a promotion whereby a user who tags a certain number of filmsbecomes eligible for a drawing to win a new vehicle.

This method of advertising in the embodiments has several benefits,including 1) it entices users to watch certain films, or to watch thefilm more than once, 2) it causes the user to “interact” with the film,and to pay attention to different aspects of it, including commercialaspects, more than would a passive observer viewing for entertainmentonly, 3) it spreads “tags” among social networks, 4) it encourages usersto participate in an advertising methodology by offering rewards, 5) itcan be low expense for the advertiser compared to more traditionaladvertising channels like print, television, or radio, 6) it allowsco-branding, 7) it is fun for the user, 8) it is an opt-in, voluntaryinteraction in which the individual elects to participate (eliminatingforced commercial advertising, for example as a bumper multimediadisplayed before the multimedia a user desires to view, which isdifficult or impossible to opt out of), 9) it leverages the efforts of amultiplicity of users to spread brand presence, and 10) it has thepotential to drive consumer behavior by utilizing the psychologicalphenomenon of social proof.

Films and television programs are replete with product placement, andthe disclosure of the present invention provides a convenient, fun wayfor consumers to “interact” with the multimedia by brand-tagging byusing mobile computing devices. Brand-tagging also alleviates challengesto standard advertising, as more DVR users consciously skip overtadvertisements forcibly inserted into a broadcast. With streaming media,users dislike a bumper ad being forced upon them before being able toview the content they elected to view, as might be found with increasingfrequency in Youtube, Hulu, Justin.tv streams and many others.

By way of example, the following are ripe for brand tagging withinmultimedia: automobiles, consumer electronics and computers, beverages(colas/carbonated beverages, energy drinks, adult alcoholic beverages(beer, wine, liquor)), clothing and accessories/fashion, airlines,travel destinations, tobacco, restaurants and eateries, foods andsnacks, hotels, firearms, radio/television/film/magazines, oil/energy,healthcare/drugs/hygiene, household items/soaps/detergents, and so on.

In practice, one way of implementing such a system includes: issuing aunique identifier, such as “BrandTagID,” to the owner of a brand (forexample, a randomized string of alphanumeric characters of predeterminedlength, akin to an encryption key used in the field of encryption);issuing a brand tag graphic or symbol or “BrandGraphic” (which itselfmay be a graphic, containing a hyperlink); having a user register toobtain a TaggingID (to allow secure registry of the identity of a uniqueuser, for example by a randomized string of characters of predeterminedlength, akin to an encryption key used in the field of encryption);combining the BrandTagID with the TaggingID, to allow a user a UniqueIDthat links a specific user to a specific brand; providing facility for auser to leave or deposit a UniqueID at a time t_(x), that being adiscrete time within a multimedia file having a run duration of t₀ tot_(end); providing facility for a computer system to receive a UniqueIDfrom a user; and providing facility for a computer system to credit adiscrete TaggingID (i.e. associated with a discrete human user) forhaving placed a UniqueID at a time t_(x) within a specified multimediafile or associated synchronization index, for example to earn entry intocontests or eligibility for prizes or giveaways. The system may alsoinclude ability to share a UniqueID with the social networking contactsof the individual associated with the corresponding TaggingID.

For example, User-1 may have a social network of 100 friends orcontacts. User-1 applies for a TaggingID, and receives TaggingID=User-1-abcd1234. The “Brand, Inc.” sporting goods company applies fora BrandTagID, and receives BrandTagID=“Brand, Inc.-xyz789,BrandGraphic=www.brandwebsite.com/graphic.gif,VariableHyperlink=www.brandwebsite.com”.

User-1 desires to brand tag for Brand, Inc., so he applies for andreceives a UniqueID, which is a mathematical function performedutilizing the TaggingID and the BrandTagID, returning a UniqueID of,say, (encrypted, signed function of) UniqueID=“abcd1234*xyz789”.

User-1 watches a motion picture and places his UniqueID at a time thatis 00:02:31.2 into the movie, when one of the main characters is seenwearing a pair of branded athletic shoes. The computer system places theBrandGraphic at that time within the synchronization index. Users in thesocial network of User-1 may elect to also view the same motion picture,opting to view the tags and comments left by User-1.

Once User-1 places a UniqueID, a report may be sent to the owner of theBrandTagID, letting the owner know the multimedia in which a tag wasplaced; the identity of the individual placing the tag, by theTaggingID; and the time/date when the tag was placed.

As a result, the owner of the BrandTagID has the ability to aggregatedata about vast numbers of instances of brand occurrence across vastnumbers of multimedia files, and it accomplishes this by leveraging thepower great numbers of disaggregate users, using mobile devices, whovolunteer their time to do it. As additional benefit, those tags areshared within the social networks of the large numbers of users.

In one embodiment, the user may use a mobile computing device to tagmultimedia that is played in a location other than on the mobilecomputing device. For example, a user watches a movie at home that isstreamed and received by an appliance, which then communicates the audioand video signals to a home theater and its components, such as an LCDtelevision and speakers. Within this movie, suppose there are twelvecompanies that have products appearing in the film and that desire usersto tag the times at which they appear. Assume each company has aBrandTagID and an associated brand graphic

The user may watch the film via his home theater system and tag brandsusing his mobile device, such as an iPad. For this example, the mobiledevice display may be partitioned into sections, wherein one brandgraphic appears in one section. The user has a touch-sensitive paletteof brands to apply to the multimedia. When that brand (or product,place, person, etc.) appears in the multimedia, performing a gesture onthat section, for example tapping the graphic, tags that brand at thattime within the multimedia. The user may have the functionality toadjust the properties of the tag, such as the TimeIn and TimeOut toprovide a duration D for the tag. The user may select a predeterminedduration to set a tag, for example by associating the creation of a tagwith a 0.5 second duration, or the duration of the tag may be set by howlong the user performs the gesture. If the product appears on screen for10 seconds, the user may press and hold the graphic for 10 seconds, andthe software will record that action.

The user may be provided facility to tag more than one brand at once.For example, assume a film character uses an Apple iPhone for a phonecall for 10 seconds, then walks to a Ford Mustang to continue theconversation for another 20 seconds. The user may press and hold theApple brand graphic for 10 seconds and then simultaneously tap the Fordbrand graphic (inserting the default tag with a duration of 0.5 seconds)and then release the Apple brand graphic after another 20 seconds,resulting in a tag for the Apple iPhone for a duration of 30 seconds. Inanother embodiment, the mobile computing device may be used to displaythe multimedia and to perform brand tagging. For example, FIG. 38illustrates an exemplary layout of a display of a mobile device forbrand tagging in multimedia content. The header area 13 displaysinformation about the multimedia in which brand tagging is performed. Inaddition, the media area 12 shows a display of the video of themultimedia and the text area 14 shows a highlighted portion of a textcurrently being displayed. In addition, the display may include asection comprising various graphics of selected brands that the user maytag in the multimedia content. For example, when the user recognizes aparticular brand on a product, the user may simply click or select theappropriate graphic in this section. In the spirit of this disclosure,this brand tagging functionality may be used alone, or it may be usedwith other functionality disclosed.

User viewing statistics may also be implemented in the embodiments. Forexample, the user may see graphing of a number of users viewing a scenefor a given duration, offered to select from “most viewed”/“favorite”scenes, view scene recommendation or movie recommendation, for example,in the form of “viewers of this scene have also liked . . . ”

Exemplary screen shots of embodiments

For purposes of illustration, FIGS. 21-37 are provided to show exemplaryscreen shots of potential multimedia content that may be provided by theembodiments. Of note, those skilled in the art will recognize that theembodiments allow for variance of the display sections and/or may allowthe user to resize various sections as desired.

The screen shots may contain copyrighted material therein to teach andillustrate how multimedia content may be rendered or displayed on amobile computing device. All rights in any such copyrighted material,however, remain reserved by the owner of the copyrighted work.

FIG. 21 illustrates an exemplary screen shot of a mobile device for atranscript synchronized to a multimedia audio file. For example, asshown, an audio file of an oral argument and transcript for a U.S.Supreme Court case related to Microsoft versus i4i Ltd. Partnership. Inthe screen shot, an image illustrating that the content is related tothe U.S. Supreme Court is shown in the media area 12. The header area 13displays relevant identification of the case, e.g., Microsoft versus i4iLtd. Partnership. In the example shown, a control bar providing controlbuttons and progress of the multimedia content may also be provided inthe header area 13. The text area 14 shows a portion of a texttranscript of the dialogue of the deposition being displayed.

As described throughout this disclosure, the mobile computing device maycontrol display of the text transcript in the text area 14 in asynchronized manner with the video being displayed in the media area 12.For example, as shown, the text at line 09 is highlighted to indicatethe current location in the transcript relative to playback of the audiofile.

In addition, as shown, an advertisement may be provided at the bottom ofthe text area 14. The advertisement may appear in the form of one ormore banner advertisements. In addition, the advertisement may changebased on time, the content of the transcript being shown, etc.Furthermore, a scrub-bar may also be provided at the bottom of thedisplay on the mobile computing device.

FIG. 22 illustrates an exemplary screen shot of a mobile device playingmultimedia content and a transcript for education material from anorganization and a listing of other courses with multimedia content. Forexample, as shown, multimedia content for classes or lectures by the ABAmay be offered to the user.

In the screen shot, an image indicating the “Patent Reform Act of2009/2010: Effects on Small Businesses and Startups” from the ABA isshown in the media area 12. The header area 13 displays relevantidentification of the lecture, such as the product code, faculty, date,page count, running time, credit hours, ethics hours, sponsors, topic,and pricing. The text area 14 shows a portion of a text transcript ofthe dialogue currently being played. Finally, a scrub bar may beprovided at the bottom of the display to indicate the progress of theplayback and highlighting of text. Moreover, the screen shot may alsooffer links to other multimedia content files for other courses offeredby the institution.

FIG. 23 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript. In particular, the mobile device isshowing a deposition video and transcript of a deposition of KevinAnderson on May 10, 2011. The header area 13 displays a court associatedwith the deposition, a civil action number associated with thedeposition, a name of the deponent in the deposition, a location of thedeposition, and the date/time of the deposition. In addition, the mediaarea 12 shows a display of the video of the deposition and the text area14 shows a highlighted portion of a text transcript of the dialogue ofthe deposition being displayed. As also shown, the transcript may havean exhibit linked to one or more lines of text, as indicated by an icon,such as the paper clip icon adjacent to lines 02-06 shown in FIG. 23.

As also shown, the transcript may have an exhibit linked to one or morelines of text, as indicated by an icon, such as the paper clip iconadjacent to lines 02-06 shown in FIG. 24. Furthermore, a commentssection may also be provided in a section of the display on the mobilecomputing device. The comments section, for example, allows a user or agroup of users to record comments about the testimony.

FIG. 25 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript accompanied with a section of thedisplay for showing errata and a thumbnail of an attachment to thetranscript. In particular, the mobile device is showing a depositionvideo and transcript of a deposition of Kevin Anderson on May 10, 2011.

As also shown, the transcript may have an exhibit linked to one or morelines of text, as indicated by an icon, such as the paper clip iconadjacent to lines 02-06 shown in FIG. 25. Furthermore, an errata sectionmay also be provided in a section of the display on the mobile computingdevice. The errata section, for example, allows a user or a group ofusers to record various errata of the transcript. In the example shownin FIG. 25, Kevin Anderson on May 23, 2011 changed the word “something”to read—nothing—on line 12. Of note, the word “something” has beenchanged to be shown as strikethrough text and the word—nothing—has beeninserted in a distinctive manner. For example, as shown, theword—nothing—may be italicized, underlined, or may be displayed with adifferent font, color, or size.

FIG. 26 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a pop-up window showing a linkedexhibit. In particular, the mobile device is showing a deposition videoand transcript of a deposition of Kevin Anderson on May 10, 2011. Asalso shown, the transcript may have an exhibit linked to one or morelines of text, such as an issued U.S. patent. In the example shown, theuser has selected or click on the linked exhibit's thumbnail, which hasresulted in a pop-up window to display the linked exhibit, i.e., U.S.Pat. No. 6,714,091.

FIG. 27 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a section for exhibits linked tothe deposition. In particular, the mobile device is showing a depositionvideo and transcript of a deposition of Kevin Anderson on May 10, 2011.As also shown, the mobile device may provide a listing of availableexhibits associated with the case or the deposition. Furthermore, insome embodiments, a cost may be associated with one or more theexhibits. For example, as shown in FIG. 26, a user may buy or purchasethe set of exhibits by selecting or clicking on the “Buy” control buttondisplayed on the mobile device.

FIG. 28 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a section showing objections toportions of the testimony by the deponent. In particular, the mobiledevice is showing a deposition video and transcript of a deposition ofKevin Anderson on May 10, 2011. As also shown, the transcript may havean exhibit linked to one or more lines of text, such as drawing from aU.S. patent. In addition, the mobile computing device may provide asection of the display for objections. For example, as shown, in FIG.27, the court overruled an objection by the defendant related to lines7-13.

FIG. 29 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a summary section on the display.In particular, the mobile device is showing a deposition video andtranscript of a deposition of Kevin Anderson on May 10, 2011. As alsoshown, the display may comprise a summary section. The summary sectionmay indicate various items, such as the Plaintiff's designations, theDefendant's designations, runtime, locations, and locations of errata inthe transcript.

FIG. 30 illustrates an exemplary screen shot of a mobile device for afiltered deposition video and transcript with a summary section on thedisplay. In particular, the mobile device is showing a deposition videoand transcript of a deposition of Kevin Anderson on May 10, 2011. Asalso shown, the transcript may be filtered to show only requestedportions. In the example shown in FIG. 29, a thick black line at line 12indicates that text between lines 13-16 has been filtered from thedisplay. As also shown, a video sequencing bar with a location indicatormay be provided above the scrub bar to show relative locations of therequested portions of the deposition.

FIG. 31 illustrates an exemplary screen shot of a mobile device for adeposition video and transcript with a section listing witnesses. Inparticular, the mobile device is showing a deposition video andtranscript of a deposition of Kevin Anderson on May 10, 2011. As alsoshown, the display may provide links to other witnesses of the case. Insome instances, a cost may be associated with retrieving a witnesstestimony. For example, as shown in FIG. 30, the testimony for MarthaCampbell requires a purchase.

FIG. 32 illustrates an exemplary screen shot of a mobile device playinga news story having a video clip and corresponding text. For example, asshown, a news story from FoxNews regarding the royal wedding is shown.

In the screen shot, an image from the royal wedding is shown in themedia area 12. The header area 13 displays relevant identification ofthe story, such as the headline and tagline. The text area 14 shows aportion of a text transcript of the dialogue currently being played. Inaddition, the text area 14 may comprise a link to other content, such asa video of “William and Kate Exchanging Vows.” Furthermore, certainportions of text may have a linked exhibit or attachment, as indicatedby a paper clip icon shown adjacent to the relevant portion of the text.Finally, a scrub bar may be provided at the bottom of the display toindicate the progress of the playback and highlighting of text.

FIG. 33 illustrates an exemplary screen shot of a mobile device playingmultimedia content with a transcript and a section listing other filesavailable to the user. For example, as shown, multimedia content for“The Glenn Beck Program” may be offered to the user.

In the screen shot, an image indicating the Glenn Beck Program is shownin the media area 12. The header area 13 displays relevantidentification of the story, such as the program's topic, date, anddescription. The text area 14 shows a portion of a text transcript ofthe dialogue currently being played. Finally, a scrub bar may beprovided at the bottom of the display to indicate the progress of theplayback and highlighting of text. Moreover, the screen shot may alsooffer links to other multimedia content files for the program, such asprograms from different months.

FIG. 34 illustrates an exemplary screen shot of a mobile device playingmultimedia content with a transcript from an educational institution.For example, as shown, multimedia content from the Kellogg SchoolManagement may be provided to the user.

In the screen shot, an image indicating the speaker as Professor MichaelMazzeo is shown in the media area 12. The header area 13 displaysrelevant identification of the speech or lecture, such as the title,speaker, and date. The text area 14 shows a portion of a text transcriptof the dialogue currently being played. Finally, a scrub bar may beprovided at the bottom of the display to indicate the progress of theplayback and highlighting of text.

FIG. 35 illustrates an exemplary screen shot of a mobile device playingmultimedia content with a transcript from a musical group and sectionlisting other available content from the musical group. For example, asshown, multimedia content, such as the song “Abbey Road” by the Beatlesmay be offered to the user.

In the screen shot, an image indicating the song is shown in the mediaarea 12. The header area 13 displays relevant identification of thesong, such as the title, year, and background. The text area 14 shows aportion of a text transcript of the dialogue currently being played.Finally, a scrub bar may be provided at the bottom of the display toindicate the progress of the playback and highlighting of text.Moreover, the screen shot may also offer links to other multimediacontent files for the program, such as different songs or albums also bythe Beatles or other artists.

FIG. 36 illustrates an exemplary screen shot of a mobile device playingmultimedia content and a transcript for an audio book. For example, asshown, an audio file for the book “Unbroken” by Laura Hillenbrand isshown. In the screen shot, an image illustrating that the content isrelated to the book is shown in the media area 12. The header area 13displays relevant identification of book, such as the title, author,pages, publisher, and price. In the example shown, a control barproviding control buttons and progress of the multimedia content mayalso be provided in the header area 13. The text area 14 shows a portionof a text transcript of the dialogue of the audio currently beingplayed.

As described throughout this disclosure, the mobile computing device maycontrol display of the text transcript in the text area 14 in asynchronized manner with the audio playback. Furthermore, a scrub-barmay also be provided at the bottom of the display on the mobilecomputing device.

FIG. 37 illustrates an exemplary screen shot of a mobile device playinga movie and a transcript with a section for navigating through themovie's content.

Titles, headers and section divisions within this disclosure are meantas navigational aids and are not meant to limit the scope of thedisclosure. While multiple implementations have been illustrated anddescribed in detail, it should be understood that various modificationsin system and method design and details of construction are possiblewithout departing from the spirit and scope of the disclosure.

Use of Barcode and a Mobile Computing Device to Retrieve TextSynchronized Multimedia

In yet another example of a printed transcript, a bar code, a 2D barcode or a Quick Response Code (or “OR Code”) may be placed at the top ofa page, in any margin on the page, or adjacent to a line or range oflines of a transcription of multimedia. A barcode may refer to any formof bar codes, QR codes, lines, dots, dashes, hieroglyphs, patterns,picture identifiers, color permutations, magnetic particle deposition,special character or numeric font usage, logos, faces, or combinationsthereof, or any other similar reference that may be scanned orphotographed for machine or electronic recognition. In one exampleshown, a QR Code may be scanned by a mobile computing device (orphotographed with a camera integral to the mobile computing device andthen deciphered by software on the mobile computing device, as isunderstood in the art). The software for scanning on the mobile devicerecognizes the QR Code as a URL-type link. The URL-type link causessoftware on the mobile computing device to retrieve an electronic copyof the transcript, including with synchronized multimedia. The link maybe a deep link to the multimedia, directing the user to a point in themultimedia other than the beginning of the multimedia.

The QR code may comprise a series of alphanumeric characters, sometimesan encrypted or randomized string of alphanumeric characters. In oneembodiment, the QR code may reference a client control number. Theclient control number can be associated with the purchaser of thetranscript or other authorized recipient.

In the context of a deposition transcript, the client control number canbe associated with at least one attorney, at least one law firm, or atleast one litigant, or a preselected group of users. In this fashion, ahardcopy of the transcript can be utilized by an attorney or law firm ashas been traditionally done, i.e. by making duplicate hardcopies for useand distribution, but providing the novel and added benefit that eachrecipient of the hardcopy can conveniently access the electronictranscript and associated multimedia by scanning the QR Code with amobile computing device. In general, the client control number mayfunction to help identify the users who may be authenticated, and towhom data may properly be delivered.

The QR code may also comprise reference to the synchronization index, ora portion thereof, and the corresponding multimedia, or portion thereof.In this fashion, a user may be directed directly to the desired portionof both the transcript and the associated multimedia.

The user with the hardcopy and mobile computing device may scan the barcode, and the application may communicate with a database to determinewhether he is listed among the group of authorized users. If the user isnot authenticated, electronic access to the transcript and multimediacan be denied.

User logs may be kept to determine the identity of users of theelectronic assets, and usage and access can be monitored andconveniently reported.

As an additional benefit, it is not uncommon in the legal industry forparties to distribute hardcopies of a transcript without properly payingthe court reporting firm for the correct number of copies. For example,in multi-party or class action litigation, a group of litigants mayattempt to purchase one transcript copy from the court reporting firm,or a limited number of copies, but then share unauthorized photocopiesfor which the court reporting firm is not compensated.

In this instance, if the user scans the QR Code on the hardcopy of thetranscript, and if the user is not already a member of a group that haspaid for and can rightfully access the electronic transcript, then thesoftware or software agreement may compel the user to pay for a copy ofthe transcript. Invoicing and payment can be achieved immediately andelectronically. This is an efficient method to electronically distributetranscript copies and to provide electronic access on a mobile computingdevice to the transcription and associated multimedia.

The client control number may differ for each copy of a transcript sold.That is, each hardcopy of a transcript that is produced, for example bya court reporting firm, may be provided a unique code with a uniqueclient control number. For example, “Code 1” may identify a purchase ofa transcript copy by attorney John Doe, at Law Firm X, representingCompany A. John Doe may, for instance, seek to utilize his mobilecomputing device to see deposition testimony beginning on Page 10,Line 1. With the application running on his iPad, he can scan, orphotograph, the QR code adjacent that line on that page. When he doesso, the application communicates the client control number embedded inthe QR code. This communication is done by computer network, whichcomputer network also comprises a computer that references a userdatabase. The database is used to authenticate that the iPad belongs toJohn Doe. Once authenticated, John Doe is served data, i.e. thetranscript and associated multimedia. If the user fails theauthentication step, data delivery may be denied. In some circumstances,the authentication step may authenticate all data requesters.

John Doe may also set permissions allowing access to that transcript toother users in a group, for example other attorneys within his law firm,or certain designated individuals associated with his law firm's client,Company A. Permissions may also be limited by an administrator, forexample the court-reporting firm responsible for distributingtranscripts and allowed a reasonable fee therefor by the governing rulesof procedure. In such a circumstance, the administrator may control ormake alterations to the database that is used by the computer system inperforming the authentication.

As an example, assume that John Doe provides the hardcopy of adeposition transcript to a co-litigant at Company B. Assume that CompanyB has not purchased a copy of the transcript from the court-reportingfirm and that John Doe cannot set permissions to allow Company B accessto the transcript electronically. If a user at Company B scans the QRcode on the transcript to obtain electronic access to the transcript andassociated multimedia on his own iPad, access may be granted, in thisexample, if and only if the administrator has allowed it, i.e. the courtreporting firm is legally able to furnish a copy, and if terms for thepurchase and sale of the access have been agreed upon. The systemadministrator, or a court reporting firm utilizing the system, caninclude as part of the terms of use for the mobile device software anagreement that any asset delivery is subject to specified terms andfees, which may include a charge of a reasonable fee upon accessing atranscript or multimedia from having scanned a QR code on a hardcopy ofa transcript. Such transaction can occur as an “in app purchase”, or apurchase as is made from within the application. As one example, theApple company supports “in app purchases”, as opposed to deliveringcontent through another mechanism, such as transfer of a pdf, book,text, music, or movies via iTunes.

In this example, the user at Company B scans with his iPad the QR codeon John Doe's hardcopy transcript, Company B is immediately charged fora purchase, and access to the transcript and associated multimedia isfurnished immediately electronically. From the electronic transcript,the software enables the user to print a hardcopy of the transcript fromthe mobile computing device, or it can be formatted as a pdf, or othersuitable format, for printing or emailing. This hardcopy of Company Bwill be marked with a bar code comprising a client control number forCompany B.

In this example, although John Doe provided a hardcopy of the transcriptto a co-litigant improperly by not properly compensating the courtreporting firm, the computer server and serving software may operate toensure he doesn't inadvertently provide electronic access to what may bework product he has associated with his group's copy of the electronicassets. For example, John Doe and his law firm may have created issuecodes and made comments upon the transcript, which they did not intendto share with Company B. Because Company B is not within the set ofauthorized users in John Doe's group, any work product of John Doe'sgroup is not made available to Company B by virtue of the Company B userhaving scanned John Doe's QR code. That is, when Company B receiveselectronic access for the first time, a new, fresh user group iscreated, and the new user from Company B must indicate who is authorizedto collaborate within his own group, subject to review by a systemadministrator.

In yet another example of the printed transcript, FIG. 40 illustrateshow an issue coded transcript may be provided to a court. QR codes areplaced in the margins to allow a user of a mobile computing device toscan the code to access an electronic copy of the transcript andmultimedia at that exact position. Thus, paper filings can be made to acourt, allowing simple access to the corresponding electronic assets(transcript and multimedia) on a separate mobile computing device.

In this manner, a judge, clerk, or courthouse staff can conveniently usea mobile computing device to retrieve multimedia assets, without havingto take delivery of CDs, DVDs, or other storage media a litigant mayintend to deliver to the court. Further, an overall computer system maybe provided and managed, to ensure multimedia submitted to the court isavailable, convenient to access, able to be archived, and capable ofhaving usage tracked, for example to ensure access is provided only toauthorized users and to date the time of filing, as part of anelectronic records filing and management system.

Such use of QR codes and mobile computing devices to allow simple andeasy retrieval of specific portions of multimedia and correspondingtranscription allows novel use of mobile computing devices. Such usesmay be particularly useful in scholarly writing, news and newsdistribution, legal writing, magazines/periodicals, books, politicaldiscourse, and elsewhere.

While the examples above are presented in the context of a legalproceeding, i.e. a deposition transcript and associated multimedia, theexamples are not intended to limit the scope of the invention. Asdiscussed above, there exist many types and kinds of uses for asynchronization index with associated multimedia, being used on a mobilecomputing device.

Sharing a Transcript by Barcode

As another example, a court reporter at the conclusion of a depositionmay distribute copies of the transcript by requiring that an attorneypresent at the deposition utilize a mobile computing device to scan abarcode to retrieve the transcript. As part of the court reportingsoftware, for example, the court reporter may utilize a mobile computingdevice, such as an Apple iPad, to display a barcode to be scanned by anattorney who desires to purchase a copy of the transcript. The scan ofthe barcode evidences an offer and acceptance, forming a bindingcontract between the attorney and the court reporter to furnish a copyof a transcript. Prior to the invention, often times there existeddisputes about whether an attorney agreed to pay for a copy of thetranscript, as such orders were taken orally at the conclusion of adeposition.

After scanning the barcode, the URL-type link is opened on theattorney's mobile computing device, the transaction data is fed to acomputer system for authentication, and the transaction is recorded ashaving been made. Certain of the information from the attorney's mobilecomputing device may be communicated to the computer system, for exampleidentifying information from the mobile computing device itself, such asa UDID from an Apple iPhone or iPad and any user or owner informationthat may be retrieved. Also, the barcode may comprise informationidentifying the court reporter, for example the name of the courtreporter, the date, the deponent, the caption of the case, and so on.The court reporter may refresh the barcode on his or her mobilecomputing device for each successive attorney wanting to make a purchaseat the time of the deposition.

Sharing a Transcript by Close Proximity Communication Between MobileComputing Devices

In yet another embodiment of sharing an electronic transcript utilizinga mobile computing device, once a user has properly accessed anelectronic copy of a transcript, and associated multimedia, for exampleby having used his mobile computing device to scan a QR code printed ona photocopy of the transcript, discussed above, the authenticated usermay transmit the identity of the electronic asset, or a specificlocation within the electronic asset, to another mobile computing deviceuser by close or near-field proximity communication with another mobilecomputing device. In one example of close proximity communication, User1may “bump” his mobile computing device with another mobile computingdevice, causing communication of a client identification number and/or asynchronization index reference. User2, with his mobile computingdevice, receives the communication of the client identification numberand/or a synchronization index reference, and User2 then proceeds asthough he has scanned User1's QR code, discussed above. One example ofclose proximity communication between mobile computing devices isutilizing Near Field Communication (NFC) technology. One mobilecomputing device utilizing NFC technology is the “Samsung Galaxy S III”,which uses software referred to as “S Beam”, to allow two devices tohave their backs gently bumped together to initiate transfer of data.

Sharing a Deep Link by QR Code

In yet another embodiment, a user of a mobile computing device mayhighlight or otherwise select a range of text from a synchronizationindex, and then select an option to instruct the software to generate aQR code with information sufficient to allow a subsequent user to scanthe QR code in order to be delivered directly to that portion within themultimedia.

In FIG. 41, for example, QR Code was created, here by accessing the URLhttp:creategrcode.appspot.com/ for the sample string of text:www.sampleURL.com/ClientControl=AttorneySmith9876;Transcript=ABC123-Anderson; StartPage=14; StartLine=2; EndPage=14;EndLine=6. This string of text contains a hosting URL, herewww.sampleURL.com; a client control number, here“ClientControl=AttorneySmith9876”; a transcript reference, here“Transcript=ABC123”; and a page and line range for the designated text(and corresponding video) at page 14, lines 2-6, here shown as“StartPage=14; StartLine=2; EndPage=14; EndLine=6”.

Once that QR code is generated, it can be immediately scanned from thedisplay screen by a second mobile computing device, posted online, forexample, on one's social networking timeline as one may do using theFacebook website, emailed, printed, embedded in another document, orsaved for later use.

FIG. 41 depicts an action sequence, beginning with an issue codeddeposition transcript, for example as may be displayed on a mobilecomputing device. Here, the user intends to generate a QR code for thedesignation at page 14, lines 2-6 of the transcript. The user selectsthe function “Show QR code”, and the corresponding QR code is displayed.

FIG. 42 depicts a software user interface for displaying QR codes. Here,the rightmost quarter of the display screen has been partitioned fordisplay of QR codes. The tab “OR Code Reference” has been selected, andthe mobile computing device software automatically generates a QR codefor each designated portion of testimony. Here, a Defendant'sdesignation page 14, lines 2-6 of the transcript is shown in thecross-hatched area 42-1 covering text at page 14, lines 2-6, andcorresponding QR code is shown enlarged in the upper right corner of thedisplay screen. As discussed above, another user of a mobile computingdevice may scan that QR code to access that transcript and correspondingmultimedia at exactly that location.

Also shown in the rightmost margin is a thumbnail QR Code for thePlaintiff's designation of page 14, lines 7-12. Plaintiff's designationof page 14, lines 7-12 is depicted with a large-cross-hatched area 41-2.As the multimedia plays and the transcript scrolls, so too may thecorresponding information in the rightmost pane. The QR code may beenlarged by a gesture on the viewing screen, such as a two fingerreverse pinch, and once enlarged may be shrunken by another gesture onthe viewing screen, such as a two finger pinch.

In this example, the use of the QR code facilitates the quick and easysharing of a precise point within multimedia between one mobilecomputing device user and another. Moreover, a discrete “clip”, orpassage, or scene, or chapter, may be quickly and easily shared betweenusers, here depicted as a deposition designation.

In yet another embodiment, an aggregation of various multimedia clipscan be forwarded to a second user by use of a barcode. Suppose, forexample, a first user has issue coded a deposition transcript with fourissues, two of which are “Plaintiff's direct designations” and“Defendant's direct designations.” Suppose, further, that the first userwants to direct a second user, e.g. the court, to view on a mobilecomputing device only those sections of the multimedia that correspondto those two issue codes, to the exclusion of the other portions of thedeposition testimony. The first user creates the designations, thenselects from a GUI the command to create a barcode for the set ofdesignations. The first user's computer causes an edit decisionlist-type of file to be sent by computer network, to a server (or theedit decision list can be created by the server). This edit decisionlist, for example, may comprise a list ofStartPage.StartLine+EndPage.EndLine entries, which correspond tomultimedia times as may be found in a synchronization index. The barcodeis then generated, with characters to direct the second user to the editdecision list generated by the first user. With a mobile computingdevice, the second user is then able to scan the barcode (i.e. ifprinted and placed within a legal brief filed with the court) and toretrieve the multimedia and associated synchronization index. In thismanner, a first user can compile a set of discrete video segments thatcan be subsequently retrieved by a second user by using a barcode and amobile computing device.

Use of Barcode and a Mobile Computing Device to Pick Up Location Datafrom Video Played on a Device Other than the Mobile Computing Device.

In yet another embodiment, a user of a DVR or set top appliance, such asan AppleTV by Apple, Inc., may pause playback of pre-recorded orstreaming programming, and thereafter retrieve that exact locationwithin the multimedia using a mobile computing device. As an example,suppose a DVR receives via cable television service certain broadcastingsignals, which are recorded to a DVR, specifically the hard drive memoryresident on the DVR. On pausing, the software on the DVR may function todisplay a barcode on the viewing screen, for example a flat screenmonitor connected to the DVR by HDMI connection, whereupon a mobilecomputing device user, say using an Apple iPad, may scan the barcode,and then be delivered right to that place within the multimedia wherethe video was paused. The multimedia content, however, may be deliveredto the iPad via Wifi or wireless data connection, and the user would beable to watch the content on his mobile device. Optimally, the userexperience is a quick and seamless appearance of transference ofmultimedia. The mobile computing device acquires the barcode, havingembedded therein a client control number, and other information, toallow the multimedia delivery system to perform an authentication stepto determine whether the user or mobile computing device possessesnecessary permissions to “share” the multimedia asset.

In this example, suppose the user, a subscriber to cable televisionservices by Cox Communications in Northern Virginia, scheduled arecording of Program X on his DVR. The DVR records the program at thescheduled time. Assume the user then watches Program X, a one-hour longprogram, which was recorded on his home DVR. He stops at 37:00 into theprogram, and wishes to have his children watch the remainder of theprogram on an iPad while travelling in the car. On pausing, the user mayelect a display of a barcode. The iPad user scans the barcode. The iPadthen communicates, via data network, either wireless, 3G, 4G, LTE orsimilar, transmitting the content of the barcode to a multimediadelivery service. The multimedia delivery service then determineswhether the user is allowed access to the multimedia, then commencesdelivery of the multimedia to the mobile computing device, either bystreaming (via data network, either wireless, 3G, 4G, LTE or similar),or by streaming and caching for later playback on the mobile computingdevice.

In this example, the authentication step verifies that the mobilecomputing device user or the mobile computing device is eligible toreceive multimedia content. This authentication does not necessarilyrequire that the mobile computing device be registered to the same cabletelevision service account, for example, which is supplying multimediato the DVR.

Further, this placement of barcode for use with a mobile computingdevice may also be utilized when the multimedia is first delivered byother means as well, for example if the multimedia content resides on aBlu-ray disc and is first played therefrom. In that instance, a pause ofmultimedia played from the disc may still result in a display of abarcode, for retrieval of multimedia by a mobile computing device. Thescan of the barcode may cause, on a computer network, the seek for theidentical content for delivery at the time desired, through a deliverymechanism where the mobile computing device, or the user, is anauthorized user. For example, a Blu-ray disc, when content is paused anda barcode is subsequently displayed, may contain as a client controlnumber an indicator for the multimedia to be requested from ahierarchical list of delivery sources. For example, the system, usingthe client control number in the barcode displayed from the Blu-raydisc, may first seek to deliver the multimedia content from a multimediadelivery system controlled by the movie's production company (e.g.Disney/Pixar), then from a hierarchical number of others where the userwould have authentication to receive the identical multimedia content,like Apple, Netflix, Amazon, Hulu, Google, Facebook, or any others.

While a Blu-ray-type media player is discussed above, the multimediaplayback system is meant to include, but not to be limited to, anyappliance for reading data from a disc using an optical reader, forexample as used in reading content from CD, DVD, or Blu-ray formattedmedia; reading data from flash memory, no matter the physical formfactor, including USB thumb drives, SD memory cards, compact flash,memory sticks, video game cartridges, and the like, and; streaming mediaappliances that receive, typically wirelessly or through wired datanetwork, data transmissions from the internet.

Further, a video game console (for example as may be connected to aviewing monitor) may operate similarly, allowing a user to pause thegame, whereupon a barcode is displayed allowing the user to utilize amobile computing device to scan the barcode and be delivered to the sameor similar game on a mobile computing device. In this instance, theinformation conveyed in the barcode may prompt a gaming softwaredownload onto the mobile computing device. The barcode may also containinformation to replicate the position of the player who paused the gameon the game console. For example, the mobile computing device user mayinherit the status of the state of the gameplay, i.e. points, score,levels completed, character attributes, powers, weapons availability,lives remaining, etc. Exemplary video game consoles include Sony PS3,Nintendo Wii, desktop and laptop computers, Apple iPhone, Apple iPad,and so on.

FIG. 43 illustrates a children's movie being displayed on a televisionscreen, a scrub bar element indicating a program duration of 60:00minutes, with the program being paused at 37:00 minutes into theprogram, and a bar code (above which is shown a trademarking element,here captioned with the trademark “MobilePickup”) which functions to bescanned by a mobile computing device.

FIG. 44 illustrates incorporation into a program guide screen a scrubbar element (here indicating a program duration of 60:00 minutes, withthe program being paused at 37:00 minutes into the program), and a barcode. The program guide screen contains other elements of a summaryprogram guide screen. Using the program guide screen, and the bar code,a user may retrieve multimedia content using a mobile computing device,which authentication step determines whether the multimedia may bebroadcast to the mobile computing device as an “on-demand” multimediadelivery service. Authentication may be denied, or a barcode may not bedisplayed, for example, if an event listed in the program guide will bea live broadcast which has not been recorded or has not yet occurred, orif the event listed has been embargoed until a first broadcast date, forexample as would be the case with a weekly scheduled drama or situationcomedy.

The content programming guide can be used to retrieve a program from theguide. Retrieving a program using the barcode and the contentprogramming guide may deliver the user to the beginning of the program.In a different embodiment, the barcode may deliver the mobile computingdevice to a point in the program other than to its beginning, as isshown in FIG. 44 where a scan of the barcode is intended to allow theuser to “pick up” the multimedia at a point 37:00 into the program.

Once again, while some of the examples above are presented in thecontext of a legal proceeding, i.e. a deposition transcript andassociated multimedia, the examples are not intended to limit the scopeof the invention. As discussed above, there exist many types and kindsof uses for a synchronization index with associated multimedia, beingused on a mobile computing device.

The features and attributes of the specific embodiments disclosed abovemay be combined in different ways to form additional embodiments, all ofwhich fall within the scope of the present disclosure. Although thepresent disclosure provides certain embodiments and applications, otherembodiments that are apparent to those of ordinary skill in the art,including embodiments, which do not provide all of the features andadvantages set forth herein, are also within the scope of thisdisclosure. Accordingly, the scope of the present disclosure is intendedto be defined only by reference to the appended claims.

What is claimed is:
 1. A transcript comprising: at least one barcode;wherein said at least one barcode comprises data selected from the groupcomprising a machine readable client control number and a reference to asynchronization index; wherein said barcode is adapted to be recognizedby software operating on a mobile computing device; and wherein saidmobile computing device may be used to retrieve multimedia by a processinitiated by reading said barcode.
 2. The transcript of claim 1, whereinsaid transcript is printed and the barcode is printed thereon.
 3. Amethod for accessing a text transcription and associated multimediautilizing a mobile computing device configured by software code to reada barcode, said method comprising: reading, by the mobile computingdevice, a barcode; authenticating the mobile computing device based oncommunication with a first computer; and delivering data comprising asynchronization index and associated multimedia, after said barcodereading step and said authenticating the mobile computing device step.4. The method of claim 3, wherein said barcode comprises data selectedfrom the group comprising a client control number and a reference to asynchronization index.
 5. The method of claim 3, wherein said barcodecomprises data selected from the group comprising a client controlnumber and a reference to a synchronization index; whereinauthentication of the mobile computing device comprises communication ofa client control number by the mobile computing device to the firstcomputer, wherein said client control number is utilized to determinewhether to authenticate; and wherein said determination whether toauthenticate causes the delivery of, or prevents the delivery of, asynchronization index and associated multimedia to the mobile computingdevice.
 6. The method of claim 3, wherein said barcode comprises areference to a synchronization index; wherein said reference to asynchronization index is used to direct mobile computing device softwareto display multimedia and an associated synchronization index; andwherein said barcode comprises a deep link, said deep link causingplayback of multimedia at a point in the multimedia other than itsbeginning.
 7. The method of claim 3, wherein said synchronization indexcomprises a transcription of a legal proceeding.
 8. A method ofdelivering transcripts, said method comprising: providing a transcriptcomprising a barcode for scanning with a mobile computing device;scanning said barcode with a mobile computing device; and after saidscanning step, initiating a transaction for payment based on informationfrom said barcode.
 9. The method of claim 8, further comprising the stepof authenticating a user, said authenticating step occurring after saidscanning step.
 10. The method of claim 8, further comprising delivery tothe mobile computing device of data selected from the group comprising atranscript, a synchronization index, and multimedia .
 11. The method ofclaim 8, wherein said transaction for payment comprises payment fordelivering data to a mobile computing device, said data selected fromthe group comprising a transcript, a synchronization index, andmultimedia.
 12. The method of claim 8, wherein said transcript is atranscription of a legal proceeding.
 13. The method of claim 10, furthercomprising: converting transcript data to a format for printing, saidformat for printing comprising one transcript page per printed page, ormore than one transcript page per printed page.
 14. A method of printingtranscripts, said method comprising: printing a transcript pagecomprising text and at least one barcode, wherein said barcode indicatesa client control number for retrieving with a mobile computing device asynchronization index and corresponding multimedia.
 15. The method ofclaim 14, further comprising retrieving multimedia using a mobilecomputing device and the barcode.
 16. The method of claim 14, whereinsaid transcript is a transcription from a legal proceeding.
 17. A methodof sharing between a first computing device and a mobile computingdevice a deep link to multimedia, the method comprising: displaying, ona first computing device, a barcode that can be interpreted by softwarecode running on the mobile computing device; and retrieving, based oninformation provided in the barcode, a synchronization index andcorresponding multimedia with the mobile computing device.
 18. A methodof sharing a link to multimedia for retrieval of multimedia by a mobilecomputing device, said method comprising: displaying a barcode that canbe interpreted by software code running on the mobile computing device;and retrieving, based on information provided in the barcode, asynchronization index and corresponding multimedia with the mobilecomputing device.
 19. A method of sharing multimedia location data, saidmethod comprising: providing a first multimedia delivery system;providing a multimedia receiver, said multimedia receiver being capableof playing multimedia; wherein said multimedia receiver comprisesexecutable program code to stop or pause multimedia and to displaybarcode; and and wherein said barcode may be read by a mobile computingdevice to facilitate delivery of multimedia to the mobile computingdevice.
 20. The method of claim 19, wherein said first multimediadelivery system comprises one selected from the group comprising cabletelevision system, satellite television system, fiber optic multimediadelivery system, wireless multimedia delivery system, and internet-basedmultimedia delivery system.
 21. The method of claim 19, furthercomprising authenticating the mobile computing device.
 22. The method ofclaim 21, wherein authenticating the mobile computing device comprises asearch for multimedia from at least one multimedia delivery system. 23.The method of claim 21, wherein authenticating the mobile computingdevice comprises a search for multimedia from at least one multimediadelivery system; and wherein said at least one multimedia deliverysystem comprises one selected from the group comprising cable televisionsystem, satellite television system, fiber optic multimedia deliverysystem, wireless multimedia delivery system, and internet-basedmultimedia delivery system.
 24. A system for sharing multimedia locationdata, said system comprising: a first multimedia delivery system; amultimedia receiver, said multimedia receiver configured for playingmultimedia; wherein said multimedia receiver comprises executableprogram code to stop or pause multimedia and to display barcode; andwherein said barcode may be read by a mobile computing device tofacilitate delivery of multimedia to the mobile computing device. 25.The system of claim 24, wherein said first multimedia delivery systemcomprises one selected from the group comprising cable televisionsystem, satellite television system, fiber optic multimedia deliverysystem, wireless multimedia delivery system, and internet-basedmultimedia delivery system.
 26. The system of claim 24, wherein saidbarcode comprises a client control number.
 27. A system for sharingmultimedia location data, said system comprising: a first multimediaplayback system; wherein said multimedia playback system comprisesexecutable program code to stop or pause multimedia and to displaybarcode; and wherein said barcode may be read by a mobile computingdevice to facilitate delivery of multimedia to the mobile computingdevice.
 28. The system of claim 27, wherein said delivery of multimediato the mobile computing device occurs by data transfer from a multimediadelivery system.
 29. The system of claim 27, wherein said firstmultimedia playback system comprises at least one data reading meansselected from the group comprising a flash memory reader, a hard drive,a streaming media receiver, and an optical reader for reading data froma disc.
 30. The system of claim 27, wherein said multimedia deliverysystem comprises one selected from the group comprising cable televisionsystem, satellite television system, fiber optic multimedia deliverysystem, wireless multimedia delivery system, and internet-basedmultimedia delivery system.
 31. A method of sharing multimedia locationdata, said method comprising: providing a first multimedia deliverysystem; providing a multimedia receiver, said multimedia receiver beingcapable of playing multimedia; wherein said multimedia receivercomprises executable program code to display a content programming guideand to display barcode; and and wherein said barcode may be read by amobile computing device to facilitate delivery of multimedia to themobile computing device.
 32. The method of claim 31, wherein said firstmultimedia delivery system comprises one selected from the groupcomprising cable television system, satellite television system, fiberoptic multimedia delivery system, wireless multimedia delivery system,and internet-based multimedia delivery system.
 33. The method of claim31, further comprising authenticating the mobile computing device. 34.The method of claim 33, wherein authenticating the mobile computingdevice comprises a search for multimedia from at least one multimediadelivery system.
 35. The method of claim 33, wherein authenticating themobile computing device comprises a search for multimedia from at leastone multimedia delivery system; and wherein said at least one multimediadelivery system comprises one selected from the group comprising cabletelevision system, satellite television system, fiber optic multimediadelivery system, wireless multimedia delivery system, and internet-basedmultimedia delivery system.